Having written my first Kafka producer in Go, and even added error handling to it, the next step was to write a consumer. It follows closely the pattern of Producer code I finished up with previously, using the channel-based approach for the Consumer:
I looked last time at the very bare basics of writing a Kafka producer using Go. It worked, but only with everything lined up and pointing the right way. There was no error handling of any sorts. Let’s see about fixing this now.
With the first leg of my journey with Go done (starting from a very rudimentary base), the next step for me was to bring it into my current area of interest and work - Apache Kafka.
In the previous exercise I felt my absence of a formal CompSci background with the introduction of Binary Sorted Trees, and now I am concious of it again with learning about mutex. I’d heard of them before, mostly when Oracle performance folk were talking about wait types - TIL it stands for mutual exclusion!
I’ve been playing around with the new SerDes (serialisers/deserialisers) that shipped with Confluent Platform 5.5 - Protobuf, and JSON Schema (these were added to the existing support for Avro). The serialisers (and associated Kafka Connect converters) take a payload and serialise it into bytes for sending to Kafka, and I was interested in what those bytes look like. For that I used my favourite Kafka swiss-army knife: kafkacat.
A Tour of Go : Goroutines was OK but as with some previous material I headed over to Go by example for clearer explanations.
This is based on the Picture generator from the Slices exercise.
I’m not intending to pick holes in the Tour…but it’s not helping itself ;-)
For an introductory text, it makes a ton of assumptions about the user. Here it introduces Readers, and the explanation is good—but the example code looks like this:
Like Interfaces, the Tour didn’t really do it for me on Errors either. Too absract, and not enough explanation of the code examples for my liking. It also doesn’t cover the errors package which other tutorial do. I’m not clear if that’s because the errors package isn’t used much, or the Tour focusses only on teaching the raw basics.
This page really threw me, for several reasons:
The text notes that there’s an error (so why don’t they fix it?)
The provided code doesn’t run (presumably because of the above error)
It’s not clear if this is a deliberate error to illustrate a point, or just a snafu
So far the Tour has been 🤔 and 🧐 and even 🤨 but function closures had me 🤯 …
Each of the words on the page made sense but strung together in a sentence didn’t really make any sense to me.
Implement WordCount
This is probably bread-and-butter for any seasoned programmer, but I enjoyed the simple process and satisfaction of breaking the problem down into steps to solve using what the tutorial had just covered. Sketching out the logic in pseudo-code first, I figured that I wanted to do this:
Slices made sense, until I got to Slice length and capacity. Two bits puzzled me in this code:
I’ve never used pointers before. Found plenty of good resources about what they are, e.g.
But why? It’s like explaining patiently to someone that 2+2 = 4, without really explaining why would we want to add two numbers together in the first place.
My background is not a traditional CompSci one. I studied Music at university, and managed to wangle my way into IT through various means, ending up doing what I do now with no formal training in coding, and a grab-bag of hacky programming attempts on my CV. My weapons of choice have been BBC Basic, VBA, ASP, and more recently some very unpythonic-Python. It’s got me by, but I figured recently I’d like to learn something new, and several people pointed to Go as a good option.
Kafka Connect (which is part of Apache Kafka) supports pluggable connectors, enabling you to stream data between Kafka and numerous types of system, including to mention just a few:
Databases
Message Queues
Flat files
Object stores
The appropriate plugin for the technology which you want to integrate can be found on Confluent Hub.
For whatever reason, CSV still exists as a ubiquitous data interchange format. It doesn’t get much simpler: chuck some plaintext with fields separated by commas into a file and stick .csv on the end. If you’re feeling helpful you can include a header row with field names in.
order_id,customer_id,order_total_usd,make,model,delivery_city,delivery_company,delivery_address
1,535,190899.73,Dodge,Ram Wagon B350,Sheffield,DuBuque LLC,2810 Northland Avenue
2,671,33245.53,Volkswagen,Cabriolet,Edinburgh,Bechtelar-VonRueden,1 Macpherson CrossingIn this article we’ll see how to load this CSV data into Kafka, without even needing to write any code
In v5.5 of Confluent Platform the REST Proxy added new Admin API capabilities, including functionality to list, and create, topics on your cluster.
Check out the docs here and download Confluent Platform here. The REST proxy is Confluent Community Licenced.
Question from the Confluent Community Slack group:
How can I access the data in object in an array like below using ksqlDB stream
"Total": [ { "TotalType": "Standard", "TotalAmount": 15.99 }, { "TotalType": "Old Standard", "TotalAmount": 16, " STID":56 } ]
