Not able to create storageinsights_v1.ReportConfig with python client

[nsasmal]

I am still not able to figure out how to create a test report through python client, as the below code snippet is not working for me.

client = storageinsights_v1.StorageInsightsClient()
                frequency_options = storageinsights_v1.FrequencyOptions(frequency = storageinsights_v1.FrequencyOptions.Frequency.DAILY)
                csv_options = storageinsights_v1.CSVOptions(record_separator = ",")
                create_time = timestamp_pb2.Timestamp()
                update_time = timestamp_pb2.Timestamp()
                display_name = "test-inv-report"
                report_config = storageinsights_v1.ReportConfig(name="projects/<my-project-name>/locations/us-east1/reportConfigs/test-config-1", create_time=create_time, update_time=update_time, frequency_options=frequency_options, csv_options=csv_options, display_name=display_name)
                # Initialize request argument(s)
                request = storageinsights_v1.CreateReportConfigRequest(
                    parent="<my-project-name>",
                    report_config=report_config
                )
                # Make the request
                response = client.create_report_config(request=request)

I am getting the below error and not able to fix it with different combinations of parameters that I am trying:

File "/Users/test1/Documents/storage-insight-poc/si-env/lib/python3.11/site-packages/google/cloud/storageinsights_v1/services/storage_insights/client.py", line 795, in create_report_config
    response = rpc(
               ^^^^
  File "/Users/test1/Documents/storage-insight-poc/si-env/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py", line 113, in _call_
    return wrapped_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/test1/Documents/storage-insight-poc/si-env/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InvalidArgument: 400 Request contains an invalid argument.

[parthea]

Hi @nsasmal,

I apologize for the delayed response.Are you receiving additional error messages other than google.api_core.exceptions.InvalidArgument: 400 Request contains an invalid argument.? Usually there is more information to mention specifically which argument is invalid. I was able to get a successful response using the code below but it took a little bit of trial and error. I've provided this feedback to the API team to see if the documentation can be improved to clarify the things that I've noted (Googlers see b/292154064).

A few things to note:

• start_date and end_date arguments need to be set in storageinsights_v1.FrequencyOptions
• start_date must be at least one day after the current UTC date
• create_time and update_time are output only fields and should not be set.
• record_separator argument of CSVOptions must be either \n or \r\n. I modified the code to use delimiter instead. You may need to use record_separator as well depending on the format. Also check whether you would like to set header_required to include a header row in the csv report.
• object_metadata_report_options is a required field for ReportConfig. See the metadata fields here.
• The format for parent should be projects/<YOUR PROJECT>/locations/us-east1, assuming you want the us-east1 location

The working code below:

from google.cloud import storageinsights_v1
from google.type import date_pb2


def sample_create_report_config():
    # Create a client
    client = storageinsights_v1.StorageInsightsClient(transport="grpc")

    frequency_options = storageinsights_v1.FrequencyOptions(
        frequency=storageinsights_v1.FrequencyOptions.Frequency.DAILY,
        start_date=date_pb2.Date(year=2023, month=7, day=22),
        end_date=date_pb2.Date(year=2023, month=7, day=31),
    )
    display_name = "test-inv-report"
    csv_options = storageinsights_v1.CSVOptions(delimiter=",")
    object_metadata_report_options = storageinsights_v1.ObjectMetadataReportOptions(
        metadata_fields=("project", "bucket", "name")
    )
    report_config = storageinsights_v1.ReportConfig(
        name="projects/<YOUR PROJECT>/locations/us-east1/reportConfigs/test-config-1",
        frequency_options=frequency_options,
        csv_options=csv_options,
        display_name=display_name,
        object_metadata_report_options=object_metadata_report_options,
    )

    # Initialize request argument(s)
    request = storageinsights_v1.CreateReportConfigRequest(
        parent="projects/<YOUR PROJECT>/locations/us-east1",
        report_config=report_config,
    )

    # Make the request
    response = client.create_report_config(request=request)

    # Show the response type
    print(type(response))

sample_create_report_config()

[parthea]

I'm going to close this issue as the code sample in #11498 (comment) is working but please feel free to open a new issue if you have additional questions.

[parthea]

It turns out that the API will return an empty report config (silent failure), instead of raising an error, if a report config cannot be created. I've updated the code snippet with additional changes needed to actually create a report config, and raise an error if a report config can't be created. I've updated the bug (b/292154064) with additional feedback for the API team so we can improve the user experience and/or documentation.

from google.cloud import storageinsights_v1
from google.type import date_pb2

project = "<YOUR PROJECT>"
bucket = "<YOUR BUCKET>" # enter bucket name without gs://
location = "us-east4"

def sample_create_report_config():
    # Create a client
    client = storageinsights_v1.StorageInsightsClient()

    storage_filters = storageinsights_v1.CloudStorageFilters(bucket=bucket)
    storage_destination_options = storageinsights_v1.types.CloudStorageDestinationOptions(bucket=bucket)
    frequency_options = storageinsights_v1.FrequencyOptions(
        frequency=storageinsights_v1.FrequencyOptions.Frequency.DAILY,
        start_date=date_pb2.Date(year=2023, month=7, day=27),
        end_date=date_pb2.Date(year=2023, month=7, day=31),
    )
    display_name = "test"
    csv_options = storageinsights_v1.CSVOptions(delimiter=",", header_required=True)
    object_metadata_report_options = storageinsights_v1.ObjectMetadataReportOptions(
        metadata_fields=("project", "bucket", "name"), storage_destination_options=storage_destination_options, storage_filters=storage_filters
    )
    report_config = storageinsights_v1.ReportConfig(
        name=f"projects/{project}/locations/{location}/reportConfigs/test-config-1",
        frequency_options=frequency_options,
        csv_options=csv_options,
        display_name=display_name,
        object_metadata_report_options=object_metadata_report_options,
    )

    # Initialize request argument(s)
    request = storageinsights_v1.CreateReportConfigRequest(
        parent=f"projects/{project}/locations/{location}",
        report_config=report_config,
    )


    # Make the request
    response = client.create_report_config(request=request)
    if not response.name:
        raise Exception("Failed to create report config")
    else:
        print(f"Created report config: {response.name}")

    # Get the report config that we created
    response = client.get_report_config(name=response.name)
    print(response)

sample_create_report_config()

[nsasmal]

@parthea ,
storage_filters = storageinsights_v1.CloudStorageFilters(bucket=bucket)
The above is pointing to a specific bucket. How to configure for the whole project for source, in stead of individual bucket?

[parthea]

@nsasmal , I don't believe it is possible based on the docs, but I will check with the API team (Googlers see b/293570925).

Inventory reports contain a list of objects and their associated metadata for a given bucket, also known as the source bucket.

[nsasmal]

@parthea , The link you provided is related to creating inventory reports using gcloud command, REST API and Console option(https://cloud.google.com/storage/docs/insights/using-inventory-reports)
With the use of python client, the source can be a project in stead of a bucket per my understanding. Please check with the team on it.

[parthea]

With the use of python client, the source can be a project in stead of a bucket per my understanding.

I've confirmed that this feature is not yet available. In the meantime, there is a possibility to list all of the buckets in a project and run reports individually on each bucket.

Regarding the comment in #11498 (comment), the following improvements have been made on the backend and in docs:

• Added explicit validation on the server side with an appropriate message when these fields are missing.
• Updated the ReportConfig proto with a link to the documentation describing various fields and validations that are associated with those fields. (see docs: Add link to documentation for ReportConfig proto fields #11562)

I'm going to close this issue but please open a new issue if you have additional questions.