Can we breathe life back into this project? #1162
Description
Hi All,
It is pretty obvious that the project has come to a bit of a halt and I wondered if there was anything that we can do to get it up and running again?
I don't know the reason why it stopped in February, maybe if we knew that we could support it in some way?
I do know that I have had multiple orgs who wanted to use .NET and Spark in Databricks but in all honestly I couldn't recommend using it as it seems the project is now dead.
What would it take to bring it up to date with both .NET 7 support and support up to Apache Spark 3.4.1?
Is Microsoft willing to invest in the project or is the community able to take it on and progress it?
I'm just hoping to start a discussion really as it is a shame so much work went into this and we were so close to being able to use .NET instead of Python or Scala but it feels frustrating that we can't use .NET.
Activity
GeorgeS2019 commentedon Sep 14, 2023
Update 30th Aug 2023
#1160
GoEddie commentedon Sep 14, 2023
but is there any way to bring it back to life?
luisquintanilla commentedon Sep 20, 2023
@GoEddie @GeorgeS2019 since the project is open-source and part of the .NET Foundation is this something the community would be interested in contributing to in order to help move it forward?
GeorgeS2019 commentedon Sep 20, 2023
@luisquintanilla We need to identify members here who are interested to maintain and merge the e.g. PR
For members here who are interested, please let @luisquintanilla know.
@msaltan PR: 3.3 and 3.4 support initially added
GoEddie commentedon Sep 20, 2023
@luisquintanilla i'm definitely interested in helping to keep the project moving forward, I stopped raising/reviewing pr's as they were not getting reviewed and merged in but if there is a committer who is available to do that or if there was an opportunity for community members to become committers then I would be interested.
luisquintanilla commentedon Sep 20, 2023
Thanks @GoEddie & @GeorgeS2019. This is definitely something for us to look into how we can unblock you and by extension the project.
dbeavon commentedon Sep 29, 2023
@luisquintanilla I can try to help. I've encountered some minor bugs that may be low-hanging fruit.
One thing that might bog us down the most is not having the means of updating the Microsoft nuget package.
https://www.nuget.org/packages/Microsoft.Spark/
Can you please explain (or give us links that explain) how the nuget packaging works for community projects, and whether it is still possible to publish new versions of it (even after Microsoft has abandoned the community)? Will someone other than Microsoft need to start publishing a different nuget?
Also I think there are portions of this project that need to be killed as the first order of business (especially if they were done on behalf of stakeholders who have left). For example, I'm pretty eager to kill all the weird cruft related to "Microsoft.Data.Analysis". That is a very minor amount of code that never worked well, and caused a lot of confusion. For example there are critical overloaded class names like "DataFrame" which are part of both namespaces! It was a bad fit for this project. Anyone who still needs to do an integration with "Microsoft.Data.Analysis" can do their own independent work to reintroduce that mess on their own. (That other project isn't even v.1 yet, in any case.)
dbeavon commentedon Sep 29, 2023
@GoEddie
Since you asked, one reason the activity on the project seems to have stopped is because of employee attrition at Microsoft. Certain full-time developers like Rahul Potharaju @rapoth and Terry Kim @imback82 have left Microsoft and started working at Databricks instead. And the critical PMs like @MikeRys have moved on to other areas within Microsoft. (Hi there Michael! Please come back!)
I am pretty disgruntled about the Synapse side of this story. In 2022 I had migrated all my projects from Databricks to Synapse where Microsoft was trying to monetize .Net for Spark. After spending several months working on this migration of my .Net projects, I encountered a relatively innocent bug that only affected Synapse (not OSS and not Databricks). I report the bug right around the time that the engineers were exiting the company. Only at the very end of an eight month support case did they say that they can no longer support .Net, and they are removing it from future versions of Synapse Analytics. It was a pretty painful experience, as you can probably imagine. (As a side the bug turned out to be just some stupid DNS configuration issue in their Bionic Beaver image, and wasn't specific to .Net. It would have affected the other language bindings as well.)
It doesn't stop there. After they removed .Net from their 3.3 runtime, this Synapse team started to deliberately spread misinformation to discourage people from using .Net on Spark at all.
See https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-33-runtime
They say this is a "project under the .NET Foundation that currently requires the .NET 3.1 library, which has reached the out-of-support status". Of course, the claim that Spark requires .Net Core 3.1 is categorically false. I'm able to run the project on .Net 6 without any problems. To add insult to injury they tell people "We recommend that users with existing workloads ... migrate to Python or Scala." I'm guessing that they are worried about the possibility that their .Net customers will just leave Synapse and find a better Spark offering. The PG's documentation doesn't even refer people back to this community, in order to pursue an alternative path forward with .Net. I find their communication to be pretty dishonest. And it almost seems like a deliberate attempt to sabotage this project.
I keep hoping we will see first-class support for .Net in Databricks, now that they have scalped some of the smart engineers from Synapse. (I do a regular google search, but it hasn't quite panned out yet.)
I had very high hopes for Synapse when they advertised their first-class .Net language bindings. There were even promotional community sessions where .Net for Spark was discussed, like here:
https://www.youtube.com/watch?v=-VpQheD-vE8
(start watching at 33:00 or so)
In any case, I am not that worried about the future. I am pretty sure .Net isn't going to die any time soon; and neither is Spark. The marriage of this pair is off to a rocky start. But they can't be kept apart forever!
GoEddie commentedon Sep 29, 2023
@dbeavon Thanks for filling in the missing details!
I completely agree about getting rid of some other the other components, probably the first thing to do is to get the core Microsoft.Spark working with the newer versions of Spark and get rid of things like the extensions etc and maybe bring them back in the future.
GeorgeS2019 commentedon Sep 29, 2023
@GoEddie
@dbeavon
@luisquintanilla
Thanks for not giving up on this project.
The fact that we are having this discussion is somehow sad!
Microsoft can not advocate the latest AI, large language model, and copilot without empowering .NET users on Spark.NET => the key and perhaps one of the very few available BIG DATA ANALYTICS pipeline for .NET community to continue their support for Microsoft's latest AI leadership.
leo-schick commentedon Oct 13, 2023
As @luisquintanilla pointed it out: The project is part of the .NET Foundation which is open source and everybody can join to be a part of the .NET success story 😃: Become a member
That Microsoft currently decided to not invest monetary ressources (e.g. paid developers( to this project should not stop this project from going forward IMO. Its just the question on how much are the community (= we/everybody) is willing to contribute and take the lead in this.
There are other projects like Mono which (I guiess) over a decade where run by the community bringin .NET to Linux before Microsoft switched to "Microsoft loves Linux" and they started actively to work together with the Mono community to merge things together. One great result of that is now .NET MAUI (I hope I get the facts right here).
But nevertheless, I think the current issue this project has is that it is missing a a maintainer who is willing to actively review/merge PRs, handle issues, publish new releases etc.
@GoEddie @dbeavon @GeorgeS2019 Can you think to be part of the success story for .NET for Apache Spark ?
GoEddie commentedon Oct 13, 2023
@leo-schick yes but im not sure what the process would be, at the moment it seems like no maintainers are active. It would be amazing if Microsoft invested paid developers in the project but even without that we, as a community, can maintain the project but have no access to anything!
leo-schick commentedon Oct 13, 2023
As it as pointed out earlier by @GeorgeS2019 in his post, a maintainer is missing. If you want to become one, please contact @luisquintanilla
GoEddie commentedon Oct 13, 2023
Thanks @leo-schick have you got any contact details for yourself or @luisquintanilla? My email is ed.elliott@outlook.com
fwaris commentedon Oct 18, 2023
I think dotnet-spark can be more liberal about which Apache Spark versions it supports. Instead of erroring out, it can just put out a warning. Apache Spark releases are more frequent, but I have found that existing dotnet-spark code works fine with newer versions of Apache Spark.
But I have to update the dotnet-spark code locally to make it work with new Spark versions (as the nuget packages don't work).
A more liberal version matching policy should reduce some of ongoing support effort required.
11 remaining items
tonyqus commentedon Jan 3, 2025
Perhaps MSFT China can take over this project. Let me check with some devs in MSFT. Mainly they should be volunteered to contribute to this project.
Since I have left MSFT China for many years, it's hard to guarantee anything here. But I can try.
@wudanzy I added you on Linkedin
GoEddie commentedon Jan 3, 2025
Thanks, it feels like we really need some governance, perhaps by joining the dotnet foundation as a first step.
What I'd really like to see is a structure put in place to stop the project being in the position it has been in as users can't rely on it as it stands.
tonyqus commentedon Jan 3, 2025
I'm afraid DNF cannot help. At least, you cannot get any new contributors from DNF.
Let's rely on MSFT or the open source community itself.
tonyqus commentedon Jan 4, 2025
I'd like to share my experience as a senior big data architect at Gridsum (NASDAQ: GSUM) during 2019 to 2020. I was the most senior architect (T4) in the Gridsum Shanghai office.
Use Case from Gridsum
We use Spark.NET to connect to the Spark cluster in our system. Gridsum creates a data visualization system based on .NET Core 3.1. We use Postgresql as the central database and pull data from other sources such as Oracle database, Hadoop cluster, Spark cluster, SQL Server, MySQL, and so on.
vsabharwal commentedon Jan 4, 2025
GoEddie commentedon Jan 4, 2025
I'd say there are a couple areas that need to be figured out:
technical direction, there are a load of issues and we need support for the current version of spark (including the upcoming 4.0 release). Other technical issues such as do we switch to support spark connect?
governance, who is it that are the new maintainers? How can we be sure that they won't disappear? I'd like to see a proper community plan put in place (voting in maintainers, open technical discussions, etc)
Is this possible?
wudanzy commentedon Jan 5, 2025
Hi @GoEddie, to be honest I don't know the answer of non-technical questions, but I can help on technical aspects. From technical perspective we faced the same challenges like you, for example we wanted to upgrade it to 3.3 and 3.5(we already did that) so it matches with our Scala offerings, we also faced some inefficiencies around the UDF and we did some improvements internally. As Vikas suggested, if you can share a lists, we can combine that with ours and move things forward. I don't know the answers to things that are out of my control, but I am eager to see this project go back to the right track.
tonyqus commentedon Jan 6, 2025
PRs to merge
Macromullet commentedon Jan 7, 2025
As I recall two things happened:
I think 1 drove 2. We were 90% of the way to delivering a major update based on this and had to abandon 3 months of work. We just assumed that was it. Even with the project being revived if Synapse or Fabric doesn’t support it I think it’s still DOA for a lot of teams.
vsabharwal commentedon Jan 7, 2025
Macromullet commentedon Jan 7, 2025
I’m hopeful but cautious. In the absence of a commitment from the Synapse team I’m very cautious. I’m curious if we’ve heard anything on that end.
wudanzy commentedon Jan 8, 2025
Hi @Macromullet , noticed that you are in Microsoft, if that's the case, you can connect with our team (@vsabharwal is our PM lead) and see how we can help you. Reviving this project can benefit all developers & users who use C# to run spark, and this is independent of the service provider.
I don't know the story on the Synapse side, but sometimes it is a chick-and-egg problem: when the community is health and there are many spark.NET users, they may review their decision.
wudanzy commentedon Jan 8, 2025
Hi @tonyqus, thanks for the list! We will review those PRs and help them proceed. Due to limited bandwidth in our team, we have to control the number of PRs under review. If you think that some PRs are much important to others, please move them to the head of the list. Thanks!
tonyqus commentedon Jan 8, 2025
[WIP] I start reviewing all the issues
Nuget package related
Enhancements
UDF-related Enhancement
CI/CD
Documentation
grazy27 commentedon Jan 8, 2025
Nice to see more people showing interest here!
Had a few conversations with .NET devs that didn't follow this route only because project was abandoned, so I'm confident this project should live.
The .NET 8 PR is ready, while .NET 9 is not LTS and, in my opinion, not critical.
Once .NET 8 is merged, I plan to add updates for
useArrow = trueIt would also be great to address .NET Interactive UDFs, as this is a key obstacle preventing curious users from proceeding with Dotnet.Spark, I'll have a look when there's some time.