At Spark AI, we build features that extract structured data from documents. Most of the time results look solid, but then you hit one field that consistently misbehaves. Location in a news article, say, where three different places are mentioned and the model has to decide which one counts.

The instinct is to add more instructions. It helps a little. But the real problem is that you’re asking the model to do two different things in one prompt: extract fields verbatim and make a nuanced judgment. Those are different tasks, and bundling them degrades performance on both.

The fix is to split the prompt. One handles the straightforward extraction, another focuses exclusively on the hard field: tighter context, clearer task, room to reason through the ambiguity. Results improve immediately.

It’s not surprising in hindsight: asking too many things at once confuses models the same way it confuses people. A focused prompt gives the model one job to do well.

This has shown up consistently to us across simple and complex tasks, smaller and larger models alike. When something’s underperforming, the first question we ask now is: what are we actually asking this prompt to do, and can it be split?

TL;DR

Building AI products at a startup requires speed over sophistication. We ditched complex evaluation frameworks for simple CSVs and custom scripts—and it’s been surprisingly effective. Here’s why simplicity wins: engineers label their own data and learn faster, we only evaluate high-impact features, and we can swap models in minutes. If you’re a nimble team building AI products, don’t let evaluation complexity slow you down. Start simple, stay flexible, and invest in rigor only where it really matters.

The Evaluation Dilemma

When you’re a lean team building an AI-heavy product, you very quickly run into a tension: you want rigor, but you don’t have the time, headcount, or appetite for heavyweight infrastructure.

At Spark, we scrape and ingest millions of documents related to large-scale energy projects and data centers. We use these documents to provide key insights like permitting regulations, social sentiment, and opposition to our customers. Initially, we developed these features solely based on cursory manual testing, but we quickly realized that in order to have a real estimate about how good our data extraction is, we needed to run evaluations.

Evaluation is the process by which you test that your AI software is performing tasks as you expect it to.

This post is about our current approach to evals: why it looks “simpler than expected,” where it’s been surprisingly effective, and how it enables us to move fast without flying blind.

Start Simple: CSVs, Not Frameworks

Our evaluation stack is intentionally unglamorous. Here’s what this looks like in practice.

At the core, we use plain CSV files to track labeled data. When we want to evaluate a task, we manually create a small but high-quality dataset ourselves, with engineers labeling data directly. These CSVs act as our golden datasets. In the example below, you can see a dataset snippet related to a location tagging task:

From there, we write lightweight, bespoke scripts that:

• Run our classification code against the dataset
• Produces another CSV file with expected/generated task-specific outputs
• Computes metrics like accuracy, precision, and recall (especially for classification problems)

That’s it. No dashboards, no complex orchestration, no dedicated evaluation service. From there, we usually review the metrics and the CSV output in order to understand what we can improve and iterate on.

We did experiment with existing evaluation frameworks early on, like popular open-source evaluation frameworks for LLM prompts. They’re powerful and have many features already included, but for our use case, they introduced more complexity. Ultimately, what matters is to evaluate the exact business logic that runs in production. It isn’t just a prompt — it may do a database lookup, custom logic, and more. Therefore, hooking others’ tools into our own code was complicated and hard to maintain.

Writing custom evaluation code turned out to be straightforward, easier to reason about, and far more flexible. For now, CSVs and a script that writes output to a console give us exactly what we need.

Engineers Doing Labeling

Our engineers do labeling here. It is a very tedious job, but there are a couple of advantages to doing it ourselves.

You learn a lot about what output you really want and the data you are actually feeding to the models. This is especially valuable for building domain expertise and customer focus.

You also learn about how the AI reasons, what it does right and wrong, allowing you to think if iterating on the prompt is what you need – or on the output schema, or something else.

For example, when building our social media data ingestion feature, our engineers discovered that the model consistently failed on labelling posts as solar, wind, or battery because we used projectType as a schema field. Posts weren’t always related to a project, but more generally about the type of energy. The model was confused by this, so we changed the prompt and the schema to call the label topic, and we got some strong improvements on the results.

This insight came from changing a manual label—no amount of automated metrics would have surfaced this nuance early on. We adjusted our prompt and schema to explicitly handle this case, improving accuracy from 40% to 70%+ just with this change.

Evaluation Is Expensive, So Be Selective

The biggest constraint isn’t tooling; it’s time. For a lean team of engineers, creating and maintaining even a 100-row labeled dataset for a single task can take 2-3 hours—time we’d rather spend shipping features.

Labeling data, even in small quantities, is expensive. Running evaluations, maintaining datasets, and interpreting results all take real effort. Doing this for every piece of extracted information would slow us down dramatically.

So we don’t.

Instead, we apply evaluations only to the highest-leverage signals in our system:

• Location tagging
• Content topic or contextual classification
• Other core signals that materially affect product behavior

For these, correctness really matters. Small drops in quality show up immediately in user experience or downstream logic.

On the other hand, there are extractions where being wrong occasionally is acceptable. For example, if we slightly mis-extract metadata from a specific Facebook post, the impact is low.

We use a simple framework: evaluate if (1) the feature directly impacts user trust or product core value, and (2) small errors have high visibility or downstream costs. Everything else can be monitored through user feedback and production logs. In those cases, we consciously choose not to invest in full evaluation pipelines.

This selectivity is critical. Evaluations are a product decision as much as a technical one.

Knowing What’s Happening in Production

One of the biggest benefits of even a simple evaluation setup is clarity.

Having labeled datasets and repeatable evaluation code gives us a concrete understanding of how our system behaves—not just an intuition. When a user complains about a misclassified document, we can add it to the dataset, tweak the prompt, and verify that our metrics stay on track.

This forces us to be explicit about what “good” means for each task. Metrics are imperfect, but they anchor conversations that would otherwise be vague and subjective.

Model Swapping Without Fear

Another major upside of our approach is how easy it makes model iteration.

We’ve built a small in-house framework that allows every part of the app to run the same query against different models. Evaluations are fast and repeatable; we can swap models—different providers, versions, or prompting strategies—and immediately see how they compare on our golden datasets.

This has been invaluable. It lets us:

• Make cost–quality trade-offs deliberately
• Respond quickly when models change upstream
• Avoid over-committing to any single vendor or approach

Crucially, this only works because evaluations are cheap to run and easy to understand.

Pragmatism Over Perfection

Our evaluation setup is not “best practice” in the abstract sense. It’s not designed to scale infinitely, and it’s not meant to impress anyone browsing our stack.

What it does do is match our current reality:

• A nimble team
• A fast-moving product
• AI systems that must be reliable in specific, high-impact areas

As Spark grows, this approach will evolve. CSVs may turn into databases, scripts into services, consoles into dashboards. But the underlying philosophy won’t change: invest in evaluation where it buys real leverage, and keep everything else as simple as possible.

If you’re just starting to build with AI, don’t be scared of evaluations—you don’t need scale or big investments. Start small; it’s already valuable.

Key Takeaways

For those just starting to build AI features, here are the principles we’d recommend:

1. Start with CSVs and simple scripts for evaluations—fight the urge to build complex infrastructure upfront.
2. Have engineers label their own data, at least initially—the learning is invaluable
3. Be ruthlessly selective about what you evaluate—not everything deserves a full pipeline.
4. Make evaluations fast and repeatable so you can experiment with confidence
5. Let your setup evolve with your needs—simple today doesn’t mean simple forever.

I’ve already written here about how I use AI for coding, but today I want to share a few lessons from a slightly different angle: building software that leverages AI.

Build your own golden dataset

If you want AI to perform a task for you, you first have to do the task yourself.

It may seem easier start writing prompts and check the results, but without a proper dataset to measure the performance you can’t have confidence if it’s working well or not.

Building such a dataset is very tedious, especially for software engineers (like me) who love to automate everything, but it is important.

For example: if you want the AI to classify documents and assign labels, you’ll need to manually label at least 100–200 documents yourself. Basically, you’re building a questions and answers set that will give you a way to measure if the AI is doing the job properly.

Without a solid dataset like this, you’re blind. You can test a few inputs but you’ll have no real confidence that it’s going to work for your users at scale.

So my recommendation is: start by building a small but good dataset, then write code that checks if the AI’s output matches what you expect. It gives you a reliable foundation to build and test against.

Ask for reasoning

When working with AI, it’s very useful to ask not just for an answer, but for a reasoning field or also known as chain of thought in the output — a little explanation of how the AI got there.

This helps in two ways:

1. it actually makes the AI reason better internally. When it generates tokens (words), it thinks better if you encourage it to explain itself.
2. it helps you debug your prompts. By reading the reasoning, you can quickly spot if the AI misunderstood what you wanted — and refine your instructions.

In my case, analyzing the reasoning output made it clear when the AI didn’t fully get my intent. Without that feedback, I would’ve been trying things out blindly.

A small tip if you want to keep things clean: you can ask the AI to format its output in JSON, or use simple HTML/XML tags to separate the real response from the reasoning part. That way, your app can easily parse and ignore the reasoning when it’s not needed.

Be lean with frameworks

Right now, there’s a huge explosion of AI frameworks. There are tools to evaluate model responses, to abstract building RAGs, to build agents… you name it.

But my advice is simple: stay lean.

Most of these frameworks are very early, and their abstractions might not survive long-term. It’s better to get closer to the metal — build your own abstraction and understand how things work at a low level.

Writing some basic evaluation scripts yourself is usually easy, and it has a huge benefit: You’ll naturally design your code so it’s easier to test and evaluate — which is critical when building anything AI-driven.

As you build, I think there are two things you absolutely want:

1. Flexibility to swap LLM models easily, depending on cost or performance.
2. Evaluations for every core part of your app, so you always know how well your AI is performing.

Conclusion

Working with AI requires a different mindset than traditional software development. Building good datasets, understanding how models reason is definitely something new I needed to learn.

Overall, I’m excited for what’s ahead in this space. The technology is evolving rapidly and will unlock lots of opportunities to create new kinds of products! 🚀

AI Tab completion

It seems it reads my mind in a certain way. When I start a code change, it will automatically suggest very relevant completions to finish it.

As you can see in the video above, this makes me much more productive and writing code is less tedious. AI understands the intention of my change and provides very relevant completions that I can simply accept. This feature also helps reduce the amount of boilerplate code I need to write myself, which is very welcome.

But I wanted also to try something different — what if AI could write most of the code for me?

Rewriting CSS of this website

I started this website a decade ago, when Bootstrap was the state-of-the-art CSS framework and Bower was the go-to tool for managing frontend code dependencies.

Many things changed since then, at certain point Bower has been retired and Bootstrap is still good but a bit bloated for what I need. I kept hanging on to it because I didn’t have time to rewrite this blog from scratch.

Recently I started a couple of projects and Tailwind looked like a much better alternative CSS framework, so I wanted to convert my blog to use it as well.

Using Cursor Agent mode mode you can ask to write code for you by explaining in plain english what you want and it performs the changes you asked across your codebase.

I used the following prompt to rewrite my Jekyll layouts:

Rewrite this layout using Tailwind instead of Bootstrap. Keep the structure similar but it does not need to be exactly the same.

The result was good enough to start with, though I had to tweak small things here and there. Still, it was remarkable that I was able to rewrite this website without knowing all the intricacies of Tailwind. I still don’t know the framework very deeply, to be honest. 🙂

Writing an iOS app

Sometimes I play card games with family and I prefer to use an app to keep track of scores — I know it sounds very nerdy! There are a ton of them in the App store but there was none I really liked. So I thought, why don’t I just write one myself?

It has been a while since I wrote any iOS code, so I thought it would be a good idea to use AI to help me.

Using Cursor Agent mode I used prompts like:

Show a list of players representing scores in a card game. For each player show the score and buttons to increase and decrease it.

When the user taps the score, show a modal view that allows to change the score via a text field input

And so forth.

Not everything worked on the first try. Here and there I had to dig into SwiftUI to understand why a layout wasn’t working as expected or why a tap didn’t trigger what the code was supposed to do.

Nevertheless, the results were remarkable. I was able to write a fully functional app in just a few days, despite having limited knowledge of Swift, SwiftUI, and SwiftData. It wasn’t just a prototype either — I successfully shipped it to the App Store!

While doing this, I also noted that AI is less knowledgeable about iOS development compared to web development. I guess this is because there is less public data available of iOS code.

Conclusion

It’s a very exciting time to be building! Turning ideas into code is much easier and faster than before. After leaving my previous job, I started coding daily and have been enjoying it much more, mainly due to Cursor’s powerful auto-completion and the ability to have AI write code for me. While I still need to double-check and sometimes fix AI-generated code manually, it has given me a tremendous productivity boost.

Desideravo aggiornarlo con un modello smart, tuttavia ciò si è rivelato un compito non così semplice a causa delle sue funzioni di controllo della temperatura dell’acqua sanitaria e della complessa integrazione con la caldaia.

Per fortuna ho scoperto che il termostato dispone di un morsetto denominato TEL, il quale è utilizzato da un accessorio venduto separatamente. Questo accessorio consente di collegare e scollegare il termostato tramite telefono.

Tuttavia, l’accessorio non risulta particolarmente utile in quanto opera esclusivamente tramite la rete telefonica tradizionale: riceve una chiamata e attiva il termostato.

Il morsetto TEL, che viene attivato, è essenzialmente un interruttore on/off. È così che mi è venuta un’idea: perché non attivare quell’interruttore con uno smart switch?

Ed ha funzionato! A seguire vi spiego come fare

Componenti e collegamento

I componenti che ho usato sono i seguenti:

• Interruttore smart Sonoff (altri marchi vanno bene pure)
• Relé, questo è necessario perché l’uscita a line out del Sonoff non sarebbe adatta come ingresso al CAR
• Un interruttore fisico, per azionare manualmente il Sonoff

Schema di collegamento dei componenti per rendere smart il termostato CARV2

In questo modo è possibile accendere e spegnere il termostato tramite l’interruttore smart. Infatti quando l’uscita del Sonoff è accesa, il relé chiude il circuito del morsetto TEL del CARV2 che permette di configurarlo in due modi:

• Imposta il riscaldamento acceso alla temperatura comfort
• Imposta il riscaldamento secondo la programmazione comfort/economy giornaliera configurata

Potete trovare tutti questi dettagli sul manuale dell’apparecchio.

Conclusioni

Il Sonoff può essere collegato ad HomeKit, Alexa o Google Home, consentendo così di creare automazioni che attivano o disattivano il termostato in base alla presenza di persone in casa, a orari predefiniti, e così via.

Anche se i termostati smart come Ecobee o Nest offrono sicuramente funzionalità più avanzate, già con questa configurazione si possono ottenere diversi vantaggi tramite le automazioni. Ad esempio, ho impostato il termostato in modo che si spenga quando usciamo di casa e si riaccenda al nostro ritorno.

Last year, I realized that letting Google own my email is a bad idea, more so if many many people do it. If you think about it, Google is able to know about every single item I buy on Amazon, or which bank accounts I use and how much money I make, just looking at the notifications I receive from such services. This is bad for my privacy and at scale is bad for everyone.

I also realized that the model of having an email protocol and an email client is broken, it limits innovation because client developers have to work on top of protocol designed decades ago. For example, using Apple Mail for Gmail does not give you push notifications for new messages if you want them, because email protocols don’t offer this use case.

I wanted a vertically integrated solution that would give me a great user experience and I want to pay for it in order to have my privacy.

There are many email providers but many of them strive to have support for legacy protocols and don’t do much more than that, their apps for example are subpar. So my focus quickly narrowed down to iCloud and Hey.

The former because I already pay for it, in order to store safely my photos, so I could get email there for free. But I thought that the interface of Mail is not going to change much for two reasons:

1. Apple Mail has to support the legacy IMAP and SMTP protocols, thus the User Experience isn’t going to change anytime soon
2. Apple then can’t leverage a vertical integration with iCloud to provide a better experience, and even if they want to do so, the benefit will be trivial because most people use Gmail anyway

Hey instead, has the same privacy focus that Apple has and on top of it, they created a vertically integrated platform. This platform allows them to build what they have today: Imbox, Screener, Feed, Paper Trail and my bet is that it will allow them to iterate at a faster speed. Also, they are a small business that does not need to compete at Apple and Google scale, so they can keep innovating as long as enough people pay for Hey.

This is where I placed my bet then, and so far I don’t regret it

In this article I’m going to show you a situation that I ran into recently. It gave me the opportunity to do a little troubleshooting in Linux, and even understand the different ways you can create new processes on Linux.

Sysdig host instrumentation

First let’s recap a little bit how the sysdig open source tool works, as well as our commercial agent that runs on top of it.

Our core piece of technology is the sysdig kernel module. It’s loaded into the kernel and it traces every system call happening on the host. For more details information you can read our previous blog post on this.

This flow of events then goes to our sysdig userspace utility that reads them, parses them and allows you to do filtering and scripting on top.For example sysdig is able to isolate all network related activity of your database, sort processes based on docker containers, or tail the logs of your webserver. You can find a cheatsheet of common tasks here.

Our commercial product uses an agent is built on top of sysdig technology. But it uses system calls for a different purpose: we aggregate system, application, container, and statsd metrics across thousands of hosts and tens of thousands of containers. These are sent to our backend service, where you chart them, analyze, setup alerts and so on.

Ok, enough background, let’s get into the story.

The tale: statsd is broken

Everything started with a customer who was not able to see statsd metrics being collected from our agent. Our agent has a statsd capture capability that should just work - no additional endpoint needed. But it didn’t work for our customer. They were using a php-fpm configuration behind nginx.

Usually we ask customers to send us a sysdig capture, which is similar to a Wireshark trace though instead of just packet info it contains rich system call data.

sysdig -w capture.scap

With it, we can troubleshoot it offline and not require ssh access to their host. But the capture didn’t have any issues - our agent was correctly parsing statsd protocol from the system calls.

I decided to simulate their environment locally on my host. I launched a setup with nginx + php-fpm using docker-compose. Our agent detected the statsd protocol and was worked fine. I was no closer to figuring out why it wasn’t working for our customer.

Then, I had a lucky breakthrough: by a chance I restarted the php-fpm engine while the agent was running and suddenly statsd wasn’t working anymore!

So I was able to replicate it, quite good starting point. I now understood that the reason we weren’t seeing statsd metrics was that php-fpm didn’t have the appropriate container metadata (namely: ID, image name and labels) attached. If statsd traffic does not come from a container our agent does not forward it to our embedded statsd server. That’s because non-containerized processes running in the host are able to reach the agent using UDP, so forwarding it would just duplicate data. A bug like this impacts sysdig open source too, so I launched it, filtered for statsd traffic and asked for container metadata (using -pc command line argument):

# sysdig -pc proc.name = php-fpm and fd.sport = 8125
5021 07:22:37.854474986 1 phpfpmalpine_php_1 (d7bb71f7610b) php-fpm (11641:7) < connect res=0 tuple=127.0.0.1:41816->127.0.0.1:8125
5022 07:22:37.854475663 1 phpfpmalpine_php_1 (d7bb71f7610b) php-fpm (11641:7) > fcntl fd=5(<4u>127.0.0.1:41816->127.0.0.1:8125) cmd=5(F_SETFL)
5023 07:22:37.854475780 1 phpfpmalpine_php_1 (d7bb71f7610b) php-fpm (11641:7) < fcntl res=0(<3t>:::9000)
5024 07:22:37.854478368 1 phpfpmalpine_php_1 (d7bb71f7610b) php-fpm (11641:7) > sendto fd=5(<4u>127.0.0.1:41816->127.0.0.1:8125) size=34 tuple=NULL
5025 07:22:37.854499400 1 phpfpmalpine_php_1 (d7bb71f7610b) php-fpm (11641:7) < sendto res=34 data=app.metrics.foo:0.38912133891213|g
5032 07:22:37.854586118 1 phpfpmalpine_php_1 (d7bb71f7610b) php-fpm (11641:7) > close fd=5(<4u>127.0.0.1:41816->127.0.0.1:8125)
5033 07:22:37.854586634 1 phpfpmalpine_php_1 (d7bb71f7610b) php-fpm (11641:7) < close res=0

Here’s what it looked like after restarting php-fm:

169694 07:22:43.494058726 3 host (host) php-fpm (11830:7) < connect res=0 tuple=127.0.0.1:40687->127.0.0.1:8125
169695 07:22:43.494059466 3 host (host) php-fpm (11830:7) > fcntl fd=5(<4u>127.0.0.1:40687->127.0.0.1:8125) cmd=5(F_SETFL)
169696 07:22:43.494059628 3 host (host) php-fpm (11830:7) < fcntl res=0(<6>)
169697 07:22:43.494062421 3 host (host) php-fpm (11830:7) > sendto fd=5(<4u>127.0.0.1:40687->127.0.0.1:8125) size=34 tuple=NULL
169698 07:22:43.494085960 3 host (host) php-fpm (11830:7) < sendto res=34 data=app.metrics.foo:0.71194379391101|g
169705 07:22:43.494119291 3 host (host) php-fpm (11830:7) > close fd=5(<4u>127.0.0.1:40687->127.0.0.1:8125)
169706 07:22:43.494119602 3 host (host) php-fpm (11830:7) < close res=0

So the container info is parsed correctly only when sysdig starts and it reads processes metadata from /proc directory but not if a process starts while sysdig or the agent is running. This didn’t happen on the capture file from the customer because it was taken in the former case.

How is a bug like this even possible? It couldn’t be unseen for so long, so I dug into the new php-fpm process pid and captured all the events related to it. Here I found something odd: the event that creates the process is not clone but fork:

sysdig -r php-fpm.scap proc.pid = 11830

134079 07:22:41.219236734 3 php-fpm (11830) < fork res=0 exe=php-fpm args= tid=11830(php-fpm) pid=11830(php-fpm) ptid=11801(php-fpm) cwd=/var/www/html fdlimit=1048576 pgft_maj=0 pgft_min=0 vm_size=39816 vm_rss=4796 vm_swap=0 comm=php-fpm cgroups=cpuset=/docker/d7bb71f7610b01172a9b87cd09ac04709d4239f2acc73f6e237adbafc30993... flags=0 uid=0 gid=0 vtid=7 vpid=7

Why fork? On Linux, clone is the system call used to create new processes. And it’s the only system call I’d seen used so far.

Doing a bit of research I found this nice answer on stackoverflow stating that actually fork is the old system call used in linux to create new processes that has been superseded by clone.

Then I went on the code we use to parse the clone event and I found that actually we were parsing cgroups info only on clone events and not fork or vfork.

Fixing it made everything work.

Conclusion

This story reminded me a quote by Justin Cormack I heard at QCon London late last year:

“Operating systems are just a kind of technical debt”

This is absolutely true: Linus Torvalds’ mantra is “never break userspace”. This makes software compatible for a decade and more, making Linux the reliable operating system we trust and love. The downside is that part of the life of a system software engineer is dealing with traces from the past of Linux history

Side note

Why php-fpm was using fork? It wasn’t the one to blame for it. Alpine linux uses muslc instead of glibc. Digging deeper, I found it uses the fork system call to implement the fork() function instead of clone, as glibc does. It’s probably because clone is more complicated to use.

Note: this blog post has been originally published on Sysdig official blog

I started from writing little features and end up to bigger ones. Step by step I found patterns better then others, and learn how to write better quality code.

I dealt with different kind of software, written by fellow juniors and seniors engineers and I saw different approaches about dealing with problems.

Code quality

A great junior engineer is very idealistic, he wants to write very clean code with blending edge tools. He spots a lot of troubles or bad practices on already written production code and wants to rewrite everything to make it better.

A senior engineer has written a lot of software, he agrees with the junior about bad written code but does not want to change it, because it works. A junior developer hates this word but it’s true. If something works, you don’t need to waste your time changing it, and the best explanation I’ve read is this:

“Well,” they say, “look at this function. It is two pages long! None of this stuff belongs in there! I don’t know what half of these API calls are for.” Yes, I know, it’s just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I’ll tell you why: those are bug fixes

The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they’ve been fixed. There’s nothing wrong with it. It doesn’t acquire bugs just by sitting around on your hard drive.

Quote from Joel Spolsky

So do we have to stay always with the same software because it works? I did two rewrite on my career, one succeeded and one not. Why? My answer now is that you need to rewrite if your software does not fit your scenario anymore: does not scale, cannot provide more features or whatever. There is a subtle balance between the bug shower you will get and the gain you will accomplish with your change. If it’s worth to do it, you will succeed, if it’s not worth you will probably fail.

Tools

A senior engineer has worked with a bunch of technologies. Sometimes it’s a trap, because he tends to deal with a problem using always the same tools, sometimes they are still good, sometimes not.

A junior engineer has a great quality: he knows nothing. He sees problems for the first time and he can find solutions from a different angle that experienced engineers don’t find. Junior engineers are amazed by new tools of any kind: languages, databases, frameworks etc.

Listening to them can led your project to new levels, fighting the attitude of staying inside the well known circle.

New projects

When comes to a new project a junior engineer starts to write everything from scratch. Seniors instead tends to start from a similar tool and modify it to fullfill required needs. Why?

The main problem is that reading software is harder than writing it, junior finds hard to read software written by others and so he prefers to start from scratch.

Conclusion

I learnt that any production software is always bad, it has some bad practices, 1000 lines of code functions, uses a database not good for the scope and more. This because when you begin a project it’s like starting a journey without knowing the destination. Software it’s like a sculpture, you start from a blank screen and begin to engraving your code. You discover step by step what your software should do and you can’t start again from scratch at any change.

The difference between a good software and a bad one is that a well written software is easier to change even if it has some bad practices inside.

It was my first experience, I joined that company for an internship and then I continued as a full-time engineer. They taught me what means beeing an engineer. I had a 9to5 job with a permanent position, some people may say they have arrived, what do you want more?

But after two years working there I started to be very stressed, every day I was losing the fun of what I was doing. I did a lot of amazing things, I have played a key role on a lot of core components of company platform. At a certain point, they gave me a task that made me very bored. I know that a job is not full of fun every day, you have to do things you love but also things you don’t like at all. So I said to myself: Luca, complete this task, the next will be more interesting. This was the motivation I used in past and worked well.

The task assigned went very bad, it was supposed to be a revolution for the platform but this time I failed, it didn’t work well. Then it was the time for holidays. Coming back home I was hoping that something changed, that I could go back to work with a good spirit and complete the task. But it was the same, as I have never been three weeks away. So I decided to to quit, starting full-time looking for something new.

A very hard decision, but I felt inside myself that it was the correct one.

Some people did not understand my decision but I received a lot of congratulations for my choice by a lot of friends and my family. I was happy for having their support and I felt that everything would go well anyway.

So I started looking around, I spent my first day off sending CVs to all the world. The day after I received a lot of responses. Very great, it was a satisfaction. I took some time to work on my side projects, it was very amazing time.

I finally accepted an offer from a startup in Silicon Valley. Starting a new job isn’t easy. After you get acquainted in a job you feel very productive, I had a lot of knowledge and I was completing a lot of tasks everyday. When you start a new job you have to start again from scratch. Anything you say, it is wrong. And it’s very frustrating. Also I changed from a work shift of 8 hours from 9 to 6, to a job with free working hours. I can work whenever I like, from wherever I want, they didn’t care. Seems easy but it’s not. I suffered a lot staying at home all day too. It’s very difficult.

After some months I started to feel better and now I can say: yes, it was definitely the correct decision to make. Now I can make a summary of what I’ve learnt:

Learning is important

On my previous job I was unhappy because I stopped learning. If you stop learning you start to get stressed. Doing new things, learn new things is one of the most important achievements that let you be happy of what you are doing.

Don’t fall in love

I enjoyed so much working with Redis, one of the smartest database I know. I felt that everything could be solved with it. It’s wrong, if you do this mistake you are biased and you cannot do a good job. Different problems need different solutions. If you are not neutral, you may not do the right choice.

Focus on tasks, not on time

Having free working hours is fantastic. It changes how you do your job, it’s more important what you do and not when. I suggest to keep a more or less 8 hours workday but split it however you want. For example I split a day in 15 pomodoros and my goal is to complete them more or less. During the morning I go to the swimming pool or spend lunch time with my girlfriend, anything; then I go back and work till I finish my day.

Break your shift

I think that working everyday from office can be stressfull, daily commute can be very expensive in terms of money or time. But it’s important for team building. Working from home it’s very great, you have more time for you and more time to work. I think the best is to mix office work with home work, to have the best of both worlds.

Working for a startup is amazing

There are different tradeoff between working for a big company or a startup. The former gives you bigger salary and a lot of benefits but you are just a pawn on a big game. The latter gives you more risk, less salary but more potential fame. On a startup you have a chance to play a key role for the success, and it’s very amazing.

As described in Wikipedia, Median filter works in this way: given a signal, output sample is the median of last N input samples, where N can be any positive integer number. Higher is N, more aggressive will be your filter.

We need a store for last samples, in a FIFO way, pushing a new one will drop an older one. Redis provides lists, which are perfect for this scope. Also we need to sort these samples in numerical order, to get every time we want the median value. Redis provides a SORT command, which does this job.

I implemented these two primitives:

• add_sample(value) - called every time to add a new sample
• get_median() - to get the median value

For the former we need to call two Redis commands:

MULTI
LPUSH <yourkey> <value>
LTRIM 0 <N>-1
EXEC

LPUSH will simply add a new sample to the list, LTRIM ensures that at least N elements will be stored, no more.

To get median value we need to call SORT and then calculate median. SORT returns all values on that list, sorted by numerical order. I have preferred to use a script, so Redis avoids returning non useful data to clients:

local list_key = KEYS[1] -- Key of samples list

-- Sort values
local sorted_values = redis.call("SORT", list_key)
local size = #sorted_values
local median = 0.0

-- Calculate median
if size % 2 == 0 then
  median = (sorted_values[size/2]+sorted_values[size/2+1]) / 2
else
  median = sorted_values[(size+1)/2]
end
-- Use tostring because median value may be floating point
return tostring(median)

Calling this script it’s easy:

EVAL <script> 1 <yourkey>

Done!