## Resource Generation Examples
### Cassandra Generate statements that support indexing JSON data into cassandra efficiently. * CQL supports type declarations - No need for recursion during write cycles - But we need a dependency-based ordering of schemas * But types can't reference themselves - Sub-objects of schemas referencing same schema require multiple declarations
### Hive Generate statements that support analysis of JSON data with explicit typing on every defined field. * Nested arrays and sub-objects - Recursive * Must traverse to field level - Add a max_depth to class * Field definitions look different at root level than nested level - Add a seperator parameter to recursion * Collisions between activity streams types and HQL reserved words - Escape all field keys
### Pig Generate statements that load JSON data with explicit typing on every defined field. * Nested arrays and sub-objects * Must traverse to field level * Arrays must be embedded within tuples - extra snippet around arrays
## Source Generation Examples
Generate java source files that interoperate seamlessly with Apache Streams jackson-based libraries. * Classes can inherit other classes, and have sub-object that reference their own type - No need to traverse while generating each class * Collisions between activity streams types and java reserved words - Allow schema to specify javaType for declarations * References may be to classes in a compile-time maven dependency - Allow schema to specify javaType for references - Generate code for referenced classes IFF they aren't on the classpath
#### Scala Generate scala source files that are inter-operable with Apache Streams and Apache Spark. * Scala Lists, Maps, primitives are subtly different from java - Generate scala-native classes in a different package - Still use Apache Streams jackson setup with addition of JacksonScalaModule * Spark SQL and DataSets require case classes, which aren't full classes - Generate case classes for schemas that are not extended within module - Generate classes for case classes to extend - Generate traits also to facilitate object casting.
## Next Steps 1. Complete unit testing on streams-plugins modules 2. Write some documentation for each plugin 3. Integration tests with generated resources and frameworks in containers 5. Merge plugins branch(es) 6. Refactor streams modules to use POJOs generated by streams-plugin-pojo 7. Publish examples demonstrating usage of plugins and applications utilizing sources / resources.
http://streams.incubator.apache.org sblackmon@apache.org [@SteveBlackmon](http://twitter.com/SteveBlackmon "@SteveBlackmon") - Technology [@PeoplePattern](http://twitter.com/PeoplePattern "@PeoplePattern")