Alex's Cave Blog

For any .NET Developer who works on the backend side and data processing, LINQ is one of the most important super power that .NET offers to handle data collections.

Ironically, some .NET developers hates it. The cause? It’s often a tool heavily misundesrtood. The common accusations are: “It’s slow”, “It’s over-engineering”, “It makes complicated simple things”, “Uses too much RAM”. Many of this complaints are commonly misconceptions about how to handle LINQ and what are the best practices to use it. Let’s check some of this misconceptions:

• “It’s Slow”. LINQ has a powerful concept called “materialization”. A LINQ query will delay data generation and memory usage (lazy evaluation) until we call a materialization predicate (.ToList(), FirstOrDefault(), .Single(), etc). This is often misunderstood, and when we try to chain a series of queries to process a data set in a careless way calling .ToList() before all the interactions are resolved, then premature materialize will occur, causing a lot of uneeded data will be loaded into memory. This gets worse when you have some thousands or more records to be processed, causing memory hogs and performance issues.
  
  But if you use materialize on your benefit and avoid calling .ToList() or similar until the right time, then the performance boost for complex data processing is a real thing. On my experience, I love to work with LINQ on scenarios where a single query isn’t enough to manipulate a data set and data modeling requires some complex manipulation. On these cases, it simplifies a lot to create clean and big data sets with complex interactions, and it’s blazing fast.
  Let’s take one example:
  
  var myExampleQuery = this.Find()
.Where(x => x.CreationDate >= someDate)
.ToList()
.Select(x => x.Name)
.Where(x => x.SomeField == filter)
.ToList();
  
  On this simple scenario, ToList() is called just before all the filters are applied, causing an intermediate materialization of data set. A lot of records that will be filtered by the second Where will be discarded for final processing, but in the meantime, those records are already on RAM wasting resources and causing memory hogs. A right use of the LINQ syntax can be:
  
  var myExampleQuery = this.Find()
.Where(x => x.CreationDate >= someDate
&& x => x.SomeField == filter)
.Select(x => x.Name)
.ToList();
  On this case, ToList() will materialize just the required data, saving RAM and increasing performance.
  
  Of course, I cannot ignore that having any kind of ORM between database engine and business layer will have an impact on the performance by itself, but the payload can be minimized with a proper use of the materializaton and some query optimizations, giving as result a more maintanable and robust software.

• “It’s over-engineering” – This is a common complain that comes when you try to use EF or LINQ in some scenarios where your application isn’t prepared for. When we start a new application, usually we decide how to design it. Here is where we decide what tools to use and how to integrate it. LINQ and EF are tools that requires some planification to be used correctly. Some of the benefits for using this tools are: code abstraction / substitution (for EF you can change store engine without change your code or do very minimal changes), separation of concerns (which makes sense when you expect that your application will grow organically), easyness on collection manipulation, among others.
  
• “It makes complicated simple things” – Sure, there is a learning curve to use LINQ and EF correctly, but it’s not too high, and it’s worth it. For people used to work with plain SQL or DAO, this involving stop thinking on specific DB / storage engine and start thinking in Entities and models. Some old-fashioned engineers (like me 🙂 ) will have some hard time to change his mindset, but with some practice you’ll start seeing the benefits of this approach.
  
  That said, I don’t say that ALL scenarios will be a good fit with this approach. Some low-level and small tools doesn’t require this at all, and using it are simply like use a cannon to kill a bug. I’m talking here about the common business-class software scenarios where we have a big monolith application that requires flexibility and ease to maintenance. For smaller scenarios there are tools like Dapper or SqlKata which can help us to integrate some of the benefits that a ORM can bring us, but with less impact on performance.
  
  And finally, LINQ can be used even when EF isn’t used. LINQ works independently as a meta-language to handle objects and collections, so it’s a very efficient tool to handle arrays, data collections in memory and programmatic object manipulation. In fact, it simplifies a lot many of these operations that can cause to us to write a lot of for / if / while sentences on other languages. So yes, I think that (again, in the right place) it simplifies a lot many operations.
  
• “Uses too much RAM” – As I mentioned on the first point, this complain surges by an incorrect / careless uses of the materialize concept and writing poor LINQ sentences. LINQ manipulate objects, so it can be used either for handling Entity Framework entities or working with in-memory collections, so it’s powerful and has many benefits (which I’ve covered previously) so this implies, again, that we need to know when to use it and how to write LINQ sentences to not affect performance.
  
  There is also things to take in mind when we use EF that can alliviate a lot the memory consumption, like switching off ObjectTrackingEnabled property when we don’t need (this property is not needed normally when we only do read-only operations, since we do not need to track changes inside an object when we edit it, so it’s better to turn it off on scenarios where we are not modifying the objects) or adjusting the concurrency settings (by default EF uses optimistic concurrency) to avoid uneeded update checks during store operations.

If you are not familiar with LINQ / EF, or you want to know more about how to improve your skills on this tools, here are some useful links that you can use to explore more:

https://www.syncfusion.com/blogs/post/8-tips-writing-best-linq-to-entities-queries.aspx

http://firstclassthoughts.co.uk/Articles/Design/LINQExtensionMethodsBestPractices.html

https://www.itprotoday.com/development-techniques-and-management/guidelines-and-best-practices-optimizing-linq-performance