Transducers

[pasviegas]

Would be really awesome if the project was transducer compatible.

I know that it would be some extra work, but I think it might be possible.

Any thoughts?

[arnaudsj]

I've had the same thoughts :)

[sorenmacbeth]

Yes, I've absolutely been thinking the same thing. I'm still reading up on transducers, but I agree it makes complete sense to make flambo compatible with them. In the meantime, gist sketches or PRs would be great while we zero in on an implementation :)

[pasviegas]

So, I think something in this direction:

; Creating a transducer
(def process-bags
  (comp (flatmap unbundle-pallet)
        (filter non-food?)
        (map label-heavy)))

; Coll
(into [] process-bags [])
; Batch 
(extend-type JavaRDD
  clojure.core.protocols/CollReduce
  (coll-reduce
    ([rdd f]
     (flambo/reduce rdd f))
    ([rdd f start]
     (flambo/fold rdd start f))))

(into [] process-bags (f/text-file sc "data.txt"))


; Channel
(chan 1 process-bags)
;Stream 
(f/createStream sc process-bags) ;??