About
Software Engineer with experience in
Building highly scalable and available…
Building highly scalable and available…
Activity
2K followers
Joseph Odanmen commented on a post
1y
I generally agree with your sentiment – Use the best tool for the job. Developers tend not to be comfortable with SQL. It just doesn't fit with everything else the developer is doing with regards to application development. Developers think in objects and data structures rather than in rows and columns 😀
However, there are instances where using SQL as the main DSL may be ill-advised even if it is available as part of the Big Data stack. For complex data extractions and transformations, SQL functions can become very limited very quickly. It is also difficult to maintain and test large SQL scripts adequately to ensure quality.
So using PySpark or Java RDD may be an overkill when ETL and data pipelining requirements are simple but as we know, things get complicated very quickly and therefore deciding to use a more robust tool from the onset may yield dividend long term.
Perhaps this is the argument some developers may have against using SQL as the primary DSL in a Big Data Enterprise Application when they say SQL is hard and confusing.
My 2 cents Mr P – Hope you are well 🙂
Experience & Education
Licenses & Certifications
Recommendations received
1 person has recommended Joseph
Join now to viewOther similar profiles
Explore collaborative articles
We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
Explore More