But when I look into jobs in that world, a lot of them require higher degrees in mathematics or science that I don't have.
Absolutely not the case mate. I'm Senior Leadership in Data Engineering and I've got 3 GCSE's and the rest spell FUDGE... Don't fret the academics.
I'd really love to work in Python. I'd like to learn Rust.
Vast majority of our work is Python and SQL (at least at my level), understanding things like Terraform / CloudFormation are really essential to picking up the Cloud Engineering aspects on what it is we do.
I've read on here comments from users saying there are a ton of opportunities in this world, but I'm not quite sure how to frame myself on a resume to make me look like a good candidate. My education isn't in tech, but much of my work experience is. Maybe I'm just overthinking it all. Admittedly, I've been so exhausted from work and life recently that I haven't had the time to really start applying for anything.
Worth taking some down time mate? The problem with tech that goes unspoken is that it can be bloody exhausting. It can be CONSTANT knowledge work that makes your brain burn and we never truly get the downtime that we need.
Honestly, having a look at a few courses a la Data Camp / Data Quest can help you out. Looking at Udemy for AWS/Azure/GCP (fuck GCP) can be helpful for knowledge. But it's very often about experience that we look for when recruiting.
You said you've been parsing sales data, talk to that. What have you been doing? How have you been modelling the data for additional use cases beyond what you're doing? How do you ensure reliability of the data? What monitoring have you put in place to ensure that data gets to the place its supposed to, on time and of high quality? How do you ensure lower cloud utilisation and costs (more DevOps) to ensure you aren't breaking the bank? Do you prefer Inmon or Kimball data modelling?
They're all questions to give you a rough idea of what they hell we talk about. Data Engineering is a wide remit of skills and use cases. Everything from "Streaming TBs of data in IoT" to "SFTP, Model in Kimball and provide to a BI team" and the later skills are often more missing, but more important.
Is it required to have some level of knowledge in Terraform or CloudFormation before landing the job? I pick up on things quickly and those seem like systems that would be much easier to learn when you have an actual use case. I suppose I could do a Udemy crash course just to put it on my resume. But if I’ve never worked professionally in it, the experience just seems a bit superficial.
What kind of positions should I be looking for with my skill set and level? As I said initially, often times when I search Data Engineering I get jobs with monster requirements. Not just academic, but 15 years in the field like you. People often say job postings are a wishlist, but this isn’t the difference between asking 5 years and having 3. Surely there’s a spot for someone like me who’s not entry level but not senior either.
Taking time off to search is not in the cards. But it’s also hard to find the time to self-study and do courses right now. I just have to work with what my life is. And that’s keeping this job while working on getting the next one.
With the current project I head up at work, there’s no real opportunity to take it further with reporting (beyond my current email logs and logging in the database) and modeling. We have other software that does that. This project is just ingesting client sales data, parsing and scrubbing it, and then sending it off to a third party for more action. I also grab response data from that third party, process and log that, to make sure the data we sent them processed correctly on their end. It’s a lot of fun and I wish I could do more with it. This is on my resume and it’s a project I would be leaning on heavily for getting a job in Data.
I have a follow up question. Where do you find opportunities? I hear about all these "opportunities" but don't see them.
Currently I run a "People Analytics" team for a major corp (3 + Me). I have a PhD in Mathematics and 5 years as lead dev previous to that. I would like to take a higher (or similar) paying job with remote/hybrid but I seem to get little traction.
Edit: I'd be more interested in doing actual work on interesting projects. My days are filled primarily with management and glad handing execs. I want to do math again.
Pioneer's answers are excellent, so I'm just going to build on that and try to give you advice to bridge where you are now vs. where you want to go.
Most importantly, harden your SQL skills: make sure you know how to write sprocs and views well. Be able to talk about indexing strategies. Consider doing a portfolio project to gain experience with Postgres (if you want startup roles/have good taste) or MS SQL (if you want to go corporate).
Webdev is a massive asset: in interviews, pitch the fact that you know how to make internal QA dashboards and web interfaces that can monitor data jobs, and build basic visualizations of your data for end-users.
Keep leaning in to Python where you can. Use it as an enhanced version of shell scripting to automate routine tasks and workflows.
Rust is not a good use of your time YET. Most experienced DEs will lean into Rust/Go/Java (as way to make performant, cross-platform binaries when needed) but it's nearly always just a nice-to-have for non-senior roles.
Many roles will ask you about cloud experience, but the questions will typically come from a screening call with a recruiter. Based on what you've said, your best option here is to make sure you can fully provision and configure a VPS from root using bash. Pitch your shell and sysadmin skills as a form of generalist cloud knowledge; IME, the first-line recruiters eat that particular spin up like catnip, and hiring managers are fine with it as long as you're transparent about what you don't know.
You have a great base for DE and you're selling yourself well. I don't think you'll have much difficulty landing a role if you polish up your weak points. Good luck!
Absolutely not the case mate. I'm Senior Leadership in Data Engineering and I've got 3 GCSE's and the rest spell FUDGE... Don't fret the academics.
Vast majority of our work is Python and SQL (at least at my level), understanding things like Terraform / CloudFormation are really essential to picking up the Cloud Engineering aspects on what it is we do.
Worth taking some down time mate? The problem with tech that goes unspoken is that it can be bloody exhausting. It can be CONSTANT knowledge work that makes your brain burn and we never truly get the downtime that we need.
Honestly, having a look at a few courses a la Data Camp / Data Quest can help you out. Looking at Udemy for AWS/Azure/GCP (fuck GCP) can be helpful for knowledge. But it's very often about experience that we look for when recruiting.
You said you've been parsing sales data, talk to that. What have you been doing? How have you been modelling the data for additional use cases beyond what you're doing? How do you ensure reliability of the data? What monitoring have you put in place to ensure that data gets to the place its supposed to, on time and of high quality? How do you ensure lower cloud utilisation and costs (more DevOps) to ensure you aren't breaking the bank? Do you prefer Inmon or Kimball data modelling?
They're all questions to give you a rough idea of what they hell we talk about. Data Engineering is a wide remit of skills and use cases. Everything from "Streaming TBs of data in IoT" to "SFTP, Model in Kimball and provide to a BI team" and the later skills are often more missing, but more important.
Hope that helps? Ask away mate. Happy to help.
Source: Me, Data dude for 15 years.
Thanks for all of this!
Is it required to have some level of knowledge in Terraform or CloudFormation before landing the job? I pick up on things quickly and those seem like systems that would be much easier to learn when you have an actual use case. I suppose I could do a Udemy crash course just to put it on my resume. But if I’ve never worked professionally in it, the experience just seems a bit superficial.
What kind of positions should I be looking for with my skill set and level? As I said initially, often times when I search Data Engineering I get jobs with monster requirements. Not just academic, but 15 years in the field like you. People often say job postings are a wishlist, but this isn’t the difference between asking 5 years and having 3. Surely there’s a spot for someone like me who’s not entry level but not senior either.
Taking time off to search is not in the cards. But it’s also hard to find the time to self-study and do courses right now. I just have to work with what my life is. And that’s keeping this job while working on getting the next one.
With the current project I head up at work, there’s no real opportunity to take it further with reporting (beyond my current email logs and logging in the database) and modeling. We have other software that does that. This project is just ingesting client sales data, parsing and scrubbing it, and then sending it off to a third party for more action. I also grab response data from that third party, process and log that, to make sure the data we sent them processed correctly on their end. It’s a lot of fun and I wish I could do more with it. This is on my resume and it’s a project I would be leaning on heavily for getting a job in Data.
I have a follow up question. Where do you find opportunities? I hear about all these "opportunities" but don't see them.
Currently I run a "People Analytics" team for a major corp (3 + Me). I have a PhD in Mathematics and 5 years as lead dev previous to that. I would like to take a higher (or similar) paying job with remote/hybrid but I seem to get little traction.
Edit: I'd be more interested in doing actual work on interesting projects. My days are filled primarily with management and glad handing execs. I want to do math again.
Pioneer's answers are excellent, so I'm just going to build on that and try to give you advice to bridge where you are now vs. where you want to go.
You have a great base for DE and you're selling yourself well. I don't think you'll have much difficulty landing a role if you polish up your weak points. Good luck!