Comment by Algamest on Overwrite specific CSV partitions pyspark
Can you update your question with your updated query? I'll be able to update my answer accordingly from this.
View ArticleAnswer by Algamest for Find columns to select, for spark.read(), from another...
Based on your additional information in your comment:I need this list to use as variables when creating another dataframe via jdbc (I need to make a specific select within postgresql). Is there a more...
View ArticleAnswer by Algamest for Scala Dataframe.columns - Order of columns
Can we rely on this method to get the columns in the same order as they are in dataset?YesThe definition of def columns is: /** * Returns all column names as an array. * * @group basic * @since 1.6.0...
View ArticleAnswer by Algamest for How read parquet files and only keep files that...
How can I create a dataframe by excluding the files that do not contain the column "year".First off I would advise restructuring bucket to separate these files based on their schema, or better yet have...
View ArticleAnswer by Algamest for Scala Spark Add a Column with percentage of a number...
Assuming you're starting with a Dataset of something like (updated to handle Option[Map[String, Long]]):import org.apache.spark.sql.Datasetcase class Model(id: Long, dist: Option[Map[String, Long]])val...
View ArticleAnswer by Algamest for Spark - Replace first occurrence in a string
Replacing the first occurrence isn't something I can see supported out of the box by Spark, but it is possible by combining a few functions:Spark >= 3.0.0import...
View ArticleAnswer by Algamest for How to remove all characters that start with "_" from...
As @werner pointed out in his comment, substring_index provides a simple solution to this. It is not necessary to wrap this in a call to selectExpr.Whereas @AminMal has provided a working solution...
View ArticleAnswer by Algamest for Condensing a bunch of columns into one array column...
Starting with an example DataFrame (taking your first two example date columns and adding today's date to show it'll work in the future):val df = List(...
View ArticleAnswer by Algamest for How to perform conditional join with time column in...
Here is the code necessary to join the DataFrames:I have commented the code so as to explain the logic.TL;DRimport org.apache.spark.sql.expressions.Windowval SECONDS_IN_ONE_HOUR = 60 * 60val window =...
View ArticleAnswer by Algamest for Output is not showing, spark scala
As confirmed in the comments, using the show method displays the result of spark.sql(..).Since spark.sql returns a DataFrame, calling show is the ideal way to display the data. Where you where calling...
View ArticleAnswer by Algamest for how to count field with condition by spark
It's possible to conditionally populate columns in this way, however the final output DataFrame requires an expected schema.Assuming all of the scenarios you detailed are possible in one DataFrame, I...
View ArticleAnswer by Algamest for Expand tuple into dataframe using Scala Spark
TL;DRimport org.apache.spark.sql.functions.{col, explode, monotonically_increasing_id, split}df // `split` "String1, String2" into separate values, then create a row per value using `explode`...
View ArticleAnswer by Algamest for Convert Spark SQL to Scala using Window function...
The solution is to first perform count("prod_name") in a Window which is partitioned by both "garment_group_name" and "prod_name" which is then used in windowSpec.Starting with some example data:val df...
View ArticleAnswer by Algamest for Counting filtered rows in Scala Spark SQL
As @m_vemuri pointed out in the comments, there will be some performance impacts to keeping track of what it filtered out.After some experimentation the best method I can think of is to start off with...
View ArticleAnswer by Algamest for Pyspark how to join common columns values to a list value
TL;DRimport pyspark.sql.functions as F# column to join on. This could become something like:# join_col = df1.columns[0]join_col = 'id'joinDF = df1.join(df2, join_col, "inner")cols =...
View ArticleAnswer by Algamest for Terraform S3 Bucket Object's etag keeps updating on...
To prevent an update on each apply, using the new aws_s3_object resource, you can use the output_base64sha256 attribute reference.The aws_s3_bucket_object data source is DEPRECATED and will be removed...
View ArticleAnswer by Algamest for Issue creating/accessing hive external table with s3...
For me the following s3 permissions were required:s3:ListBuckets3:GetObjects3:PutObjectI was receiving the same error and was missing s3:ListBucket.As for KMS permissions (if...
View ArticleAnswer by Algamest for Databricks auto merge schema
TL;DR: add a semicolon to the end of the separate SQL statements:set spark.databricks.delta.schema.autoMerge.enabled = true;The error is actually a more generic SQL error; the IllegalArgumentException...
View ArticleAWS Lambda scala app No public method named handle with appropriate method...
I have a scala application which I have deployed on AWS lambda.Previously my Main class signature looked like:object Main extends App {...}With this I was able to provide a handler in the AWS Lambda...
View ArticleAnswer by Algamest for IAM policy to specific S3 bucket not effective without...
TL;DR: This isn't supported by AWS.I'm trying to set up the same scenario, both for least-priveleged access as well as for providing the simplest ease of use.According to the AWS knowedlege centre (as...
View ArticleAnswer by Algamest for IAM Role for Terragrunt
Assuming the terragrunt role is for deployments: Terragrunt have a doc on this.Assuming you're using an s3 bucket for state:{"Version": "2012-10-17","Statement": [ {"Action":...
View ArticleInvalidate cloudfront cache of local files uploaded to s3 using Terraform
I originally intended to post an answer to this question, however it seemed off-topic as I had a specific use-case around invalidating local files which are uploaded to s3.I want local files which are...
View ArticleAnswer by Algamest for Invalidate cloudfront cache of local files uploaded to...
resource null_resource cache_invalidation { # prevent invalidating cache before new s3 file is uploaded depends_on = [ aws_s3_object.html_files, aws_s3_object.css_files, ] for_each =...
View ArticleCreate AWS Athena view programmatically
Can you create views in Amazon Athena? outlines how to create a view using the User Interface.I'd like to create an AWS Athena View programatically, ideally using Terraform (which calls...
View ArticleAnswer by Algamest for AWS SES - Send email with different account
Emails need to be verified in the account from which the email is being sent via SES.One solution to your scenario (which I appreciate is a few years old) is to create an IAM role (ROLE1) in ACCOUNT1...
View ArticleComment by Algamest on How to load variable in Angular with http get
Please share any relevant code instead of a screenshot by editing your Question. Fewer people are likely to reproduce your issue without having your code in a copyable form.
View ArticleComment by Algamest on I have problem with the generated file .java form web3j
To help others understand your issue and to aid in finding a solution I would recommend you provide a minimal, workable example (mwe) of the issue you are facing. It's possible that the process of...
View ArticleComment by Algamest on ec2:securitygroup Requesttag condition
I posted an answer on a similar question, which I think would be of use: stackoverflow.com/a/77948494/2743131
View ArticleComment by Algamest on Terraform loops and attached my multiple roles and policy
I would start by making use of the for_each meta argument, and not handling the special case of their being 0 instances -- the for_each will handle this. Though I'm not directly providing a solution, I...
View ArticleAnswer by Algamest for Allow ec2:CreateSecurityGroup only if request contain...
TL;DR{"Action": ["ec2:CreateTags","ec2:CreateSecurityGroup" ],"Condition": {"StringEquals": {"aws:RequestTag/CreatedBy": "a_value" } },"Effect": "Allow","Resource":...
View ArticleAnswer by Algamest for Issue provisioning bucket policy error: api error...
I believe your error comes from your principal field, which should be a block that specifies that your value is an AWS Service:resource "aws_s3_bucket_policy" "bucket_policy" { bucket =...
View ArticleAnswer by Algamest for Terraform loops and attached my multiple roles and policy
Based on the information provided, I do not see an issue with how inline policies are created since there is only 1 per role.As for the managed policies, I believe the issue is to do with the [0]...
View ArticleComment by Algamest on AWS RDS Custom Oracle instance creation fails due to...
Does this role have any policies attached?
View ArticleAnswer by Algamest for Securing IAM User Access Keys in Terraform
The pros and cons of managing IAM users and their access keys directly within Terraform, especially concerning security best practices.I agree that storing access keys in the state file, no matter how...
View ArticleComment by Algamest on Uncaught TypeError: Tn.div is not a function at...
Please share any relevant code instead of a screenshot by editing your Question. Fewer people are likely to reproduce your issue without having your code in a copyable form.
View ArticleComment by Algamest on My cloned website build on nextjs won't load up
When asking a question, people will be better able to provide help if you provide code that they can easily understand and use to reproduce the problem. This is referred to by community members as...
View ArticleAnswer by Algamest for AWS RDS Custom Oracle instance creation fails due to...
You error is telling you that you need to add all of the actions which you have already specified. Looking at Step 3 in this doc I believe the issue is with your assume_role_policy preventing RDS from...
View ArticleComment by Algamest on When I running pytest, I receive the following error...
Please share any relevant code instead of a screenshot by editing your Question. Fewer people are likely to reproduce your issue without having your code in a copyable form.
View ArticleComment by Algamest on How can I extract secrets using GitHub Actions?
For me it required changing print q to print(q)
View ArticleComment by Algamest on How to sec color of each bars of histogram in ATAS c#
Please provide your current code so that additions and changes can be suggested.
View ArticleComment by Algamest on Access Denied when trying to access S3 from AWS Lambda
Can you share the code that the Lambda is running as well as share the error message?
View Article
