2023-07-07 - Data Engineering #4 - Data Lakes - Chewing the Fat 📈

[lukecookssw]

Hey data engineers!

Let's talk about this rule

The rule has this Scenario: Imagine you work at Northwind. They have all these systems:

• GitHub/Azure DevOps – issues, commits, etc.
• Billing - Timesheets, Client invoices, etc
• CRM – Projects, skills, technologies, sales, leads, etc.
• Accounting – Business expenses, client payment receipts, payment dates, etc.
• Office automation logs – employee arrival/leave times, device usage, security alerts, etc.
• Websites – traffic, analytics, etc.

Comment below for what you think would be:

1. A cool question would you ask this data? Be creative!
2. How many systems would you be interacting with?
3. Would you need a Data Lake or Data Warehouse?

[GordonBeeming]

1. A cool question would you ask this data? Be creative!
It would be cool to have suggested rules... the way I'd see this working is that by looking proportionally to where our billing and skills on those projects are, we should have rules that highlight those same areas. Looking at the recency of rules would taken into account too, so if we've got 400 rules on .net but they are all outdated, we should fix that.

2. How many systems would you be interacting with?
4

3. Would you need a Data Lake or Data Warehouse?
It could start off using a data warehouse, although expanding on it might be easier down the line with a data lake

[drwharris]

EDIT: Updated to account for the change in question.

A cool question would you ask this data? Be creative!

How used is a feature in comparison with the effort expended in development

How many systems would you be interacting with?

Source control and the end system

Would you need a Data Lake or Data Warehouse?

Depends if any Transformation is needed. None, then a DW. else DL

OLD:
Maybe tie the github/azure stats into the "leaderboard" in SSW Times. Whos winning the PR or commit race?
Also some info on what rules are getting viewed, by who?

[brydeno]

More interestingly,

• do employees who work longer produce more?
• Do they produce code with less issues?
• Does this validate the external research around shorter working hours increasing productivity (ie the push towards a 4 day week)?

[tkapa]

cool question would you ask this data? Be creative!
Productivity metrics of teams with projects on different platforms (Gh, AzDo)

How many systems would you be interacting with?
4?

Would you need a Data Lake or Data Warehouse?
Depending on scale, I'd likely start with a Warehouse and scale up.

[matt-goldman]

A cool question would you ask this data? Be creative!

I would want to know whether we can draw any conclusions about how the intersection of all the factors that feed into this data impact productivity. It would be cool to see if we can identify for example environmental factors in the offices that affect this. We of course need to be aware that correlation does not equal causation - one obvious example is that it would be easy to look at the data and say that the more time we spend on internal products, the less money we earn (when in fact the causal relationship is the other way around). I'm more interested in what makes developers happy and comfortable. And obviously this impacts revenue and the profitability of the business. It would also be cool to see if we can identify client personas or verticals that yield the best outcomes, and this would help us focus our efforts on driving engagement in those areas, or shoring up the gaps.

How many systems would you be interacting with?

All of them.

Would you need a Data Lake or Data Warehouse?

Both. A Data Lake to aggregate all the data and feed into the analysis, and a Data Warehouse for reporting.

[ncn-ssw]

A cool question that could be asked to this data might be: "Can we predict a project's success based on historical timesheets, project data, associated skills and technologies, and correlating factors such as employee arrival/leave times, device usage, and website traffic during the project's timeframe?"

For this question, I'd be interacting with the following systems:

1. GitHub/Azure DevOps - to gather data on the issues and commits related to the project.
2. Billing - to collect timesheet data which may indicate project's progress and employee dedication.
3. CRM - for project details, associated skills and technologies.
4. Office automation logs - to analyze employee's work pattern and device usage during the project.
5. Websites - to get an understanding of the traffic and analytics related to the project or the product.

Considering the variety, volume, and the potential semi-structured nature of this data, we would likely benefit from using a Data Lake to store the raw data. Then, we could leverage a Data Warehouse to store the processed, cleaned, and structured data ready for analysis. 🤖⭐

[ozdemir-mehmet]

A cool question would you ask this data? Be creative!
How much does each PR cost, the group
How many systems would you be interacting with?
Maybe 3
Would you need a Data Lake or Data Warehouse?
DataWarehouse would always be my end goal

[jakebayliss]

A cool question would you ask this data? Be creative!

Show me SSW's commits per day over time

How many systems would you be interacting with?

2

Would you need a Data Lake or Data Warehouse?

Warehouse

[calumjs]

Question	Systems Involved	Data Lake or Data Warehouse
Can we predict the likelihood of future sales leads based on the type of projects we are currently working on and the pattern of website traffic?	CRM 📊, Websites 🌐, GitHub/Azure DevOps 💻	Data Warehouse 🏦 (Mostly structured data involved)
How does the interaction of the technologies used in projects with the skills of the developers affect the billable hours and the resulting project profitability?	CRM 📊, GitHub/Azure DevOps 💻, Billing 💵, Accounting 📔	Data Lake 🏞️ (Unstructured data like commits or issues from DevOps needs to be analyzed)
Based on the patterns in website traffic and the types of technologies/skills involved in our projects, can we predict which technologies or skills will be in high demand in the future?	CRM 📊, Websites 🌐, GitHub/Azure DevOps 💻	Data Warehouse 🏦 (Mostly structured data involved for pattern analysis and trend prediction)

[Jord-Gui]

1. A cool question would you ask this data? Be creative!
   How much money does it cost to maintain devs that don't do any work 😈

2. How many systems would you be interacting with?
   4

3. Would you need a Data Lake or Data Warehouse?
   Data Warehouse

[NadeekaJK]

Interesting stuff!!

[vladislav-kir]

1. A cool question would you ask this data? Be creative!
   Do people who spend more time in the office are more or less productive?

2. How many systems would you be interacting with?
   2 - Use Office automation logs – employee arrival/leave times and GitHub/Azure DevOps – issues, commits...

3. Would you need a Data Lake or Data Warehouse?
   Data Warehouse

[cvionnet]

Great video, it gives a better understanding of the role of Data Engineering and Data Scientist.

1. A cool question would you ask this data? Be creative!
   "What is the correlation between employee arrival/leave times and their productivity based on GitHub/Azure DevOps commit patterns?"🤖

2. How many systems would you be interacting with?
   Probably 3

3. Would you need a Data Lake or Data Warehouse?
   It depends©

• If the company just wants to show off their existing data with fancy reports, a data warehouse should do the trick
• But if the company has a bunch of data scientists who like to play around and make predictions with data, throwing in a Data Lake would be a smart move

[tombui99]

A cool question would you ask this data? Be creative!

Based on the CRM invoice and GitHub commits data, can we tell if there's a correlation of the productivity of a developer between when working internally and on client?

How many systems would you be interacting with?

3

Would you need a Data Lake or Data Warehouse?

It would be useful to have them if we plan on making fancy dashboards or make predictions

[2fernandez]

[AlexRothwell]

A cool question would you ask this data? Be creative!

Is there a relationship between PR size and delivered value?

How many systems would you be interacting with?

4

Would you need a Data Lake or Data Warehouse?

Data lake - how value is defined will change based on project, context and stakeholder - being able to explore in different ways would be useful

[hveraus]

1. A cool question would you ask this data? Be creative!
   Does people just back from holiday have higher or lower productivity than before taking the holiday. e.g. more PRs done?

2. How many systems would you be interacting with?
   Maybe 3

3. Would you need a Data Lake or Data Warehouse?
   N/A

[KristenHu]

1. A cool question would you ask this data? Be creative!

How many complaints would we receive from our clients every month🧐

3. How many systems would you be interacting with?

4 or 5 I think.

4. Would you need a Data Lake or Data Warehouse?

Data Warehouse

[AnkyChe]

1. A cool question would you ask this data? Be creative!
   What skill is easy to learn but very profitable?

2. How many systems would you be interacting with?
   3 I think

3. Would you need a Data Lake or Data Warehouse?
   Data Warehouse

[Freego1783]

Data engineering looks really promising, I'm keen to see what insights we can get out of SSW's data !

[PothieuG]

1. A cool question would you ask this data? Be creative!
   Can we use traffic from the application to create new marketting strategies ?

2. How many systems would you be interacting with?
   Three

3. Would you need a Data Lake or Data Warehouse?
   Data Warehouse

[BrookJeynes]

A cool question would you ask this data? Be creative!

Are people who work less hours more productive in the hours they work vs people who work more hours

How many systems would you be interacting with?

Maybe 3

Would you need a Data Lake or Data Warehouse?

Data Warehouse

[sethdaily]

A cool question would you ask this data? Be creative!
Does the frequency of GitHub/DevOps commits on a project correlate with client satisfaction?
How many systems would you be interacting with?
2 - GitHub/Azure for commits, and CRM for projects and client satisfaction data
Would you need a Data Lake or Data Warehouse?
I think you would need a data lake, because this would be a unique set of queries that the company probably does not have a data warehouse for (you could probably build a data warehouse that does this too).

[leahy268]

A cool question would you ask this data? Be creative!
I would be interested in productivity versus office/remote work probably correlated with customer satisfaction and developer satisfaction.

How many systems would you be interacting with?
Three

Would you need a Data Lake or Data Warehouse?
Not sure

[UlyssesMaclaren]

A cool question would you ask this data? Be creative!
What referral sources brought us our main clients?
Profit and loss per month /quarter?
Which developers get the most work done?

How many systems would you be interacting with?
All of them (6)

Would you need a Data Lake or Data Warehouse?
Data Warehouse as all the data looks structured

[tomek-i]

Which developers get the most work done?

Hmm. What do you consider as more work? The amount of PBI's completed? The amount of lines changed? Something different?

• Amount of PBI's can be misleading, I could have 10 PBI's with super small fixes like spelling vs, 1 PBI that introduced a new feature with 1000 of lines.
• Amount of lines can also be misleading because I could have 10.000 lines changed where I did a simple find & replace where I renamed a variable. vs troubleshooting for 2 days a bug until finding the root cause to be a logical error. Fixing this could be changing one line.

Perhaps the question would be, which of the work had the most value to the business? That would make sense, however there are also many scenarios that can occur that skew it. For example consider the scenario that a developer picks unassigned PBI's from the print / backlog and has a streak of picking low value PBIs?

Finding good and consistent correlation between all of these variables is the difficult part :-)

[leehawkinsssw]

1. How does the number and severity of bugs vary based on developer seniority and time of day?
2. 2-3
3. Data warehouse would suffice (data to support answering this question feels structured in nature)

[Marxoz]

A cool question would you ask this data? Be creative!
Based on the enquiries we receive through the website, what are the main technologies requested.
And how does that compare to the existing projects that we are working on. Break it up into number of projects as well as amount of $.

How many systems would you be interacting with?
3

Would you need a Data Lake or Data Warehouse?
Although a Data warehouse should be efficient, a Data lake will make more sense. As you answer the questions above, you will start building patterns and ask more questions, which will require more data. This data might only be know once you have answered the question above, so a Data Lake might be more useful in the long run

[chrisschultzssw]

A cool question would you ask this data? Be creative!

Which lead sources end up being the most profitable?
Similarly - which skills/tech are the most profitable?

How many systems would you be interacting with?

3

Would you need a Data Lake or Data Warehouse?

Data warehouse

[Hona]

A cool question would you ask this data? Be creative!

Does the device you choose to work with (Windows/Mac etc), impact any metrics of statistical significance?

How many systems would you be interacting with?

Possibly all of them

Would you need a Data Lake or Data Warehouse?

Data Warehouse.

[Levijj22]

Incredible video! What a great explanation.

[KostaMadorsky]

1. Cost per line of code 🙃
2. Gotta Pull ’Em All
3. DW