Step 4: Upload Your Catalog File to HAQM

After you've verified that your catalog file is valid and set up your AWS account, you're ready to upload your catalog file to the S3 bucket that HAQM set up for your catalog. You can upload your catalog using the AWS command line interface.

• Upload Your Catalog File
• Example Workflow
• Automating the Validation and Upload Process
• Additional Resources for AWS
• Catalog Uploading FAQ
• Troubleshooting
• Next Steps

Upload Your Catalog File

You upload your catalog files to a catalogs folder inside the S3 bucket that was set up for you. Use any S3 tool to upload your catalog file to this bucket and folder. (If you haven't yet set up your AWS account, see Step 3: Set Up Your AWS Account for Catalog Ingestion first.)

To upload a catalog file to an S3 bucket:

1. Type the following command, substituting your catalog file name for <catalog_file_name.xml> and S3 bucket name for <s3_bucket_name>. Note that the --acl bucket-owner-full-control option is required so that HAQM will be able to read the file that you upload upload, and thus ingest your catalog.

   $ aws s3api put-object --body <catalog_file_name.xml> --bucket <s3_bucket_name> --key catalogs/catalog.xml --region us-east-1 --acl bucket-owner-full-control
   

   Note: If you copy-and-paste this command into a terminal window, make sure that the -- characters paste as double-dashes and not as an n-dash. Also, make sure you set the --acl parameter with bucket-owner-full-control; otherwise, the upload will fail.

   This command successfully uploads files up to 2GB in size. If successful, the AWS CLI will display a set of tags for the VersionID, ETag, and Expiration for the file.

2. Verify that you see your catalog file in your S3 bucket. Type the following command to list all files in your S3 bucket:

   $ aws s3 ls s3://<s3_bucket_name>/catalogs/ --region us-east-1
   

   (Replace <s3_bucket_name> with your bucket name.)

   You can upload multiple catalogs to the catalog bucket. However, HAQM uses only the most recently uploaded catalog, regardless of its name. If you find a problem with your catalog, or need to update the data in the catalog file, just upload a new file.

Example Workflow

The following example command uploads a catalog file named my-catalog.xml to an S3 bucket named cdf-bucket:

$ aws s3api put-object --body my-catalog.xml --bucket cdf-bucket --key catalogs/catalog.xml --region us-east-1 --acl bucket-owner-full-control

This command returns the following sample output:

{
    "VersionId": "m_QwgKPy9RJZsWperU_LEULD1waJE2He",
    "ETag": "\"e8c38d5258ad1f3b241ae2ce347e40bc\"",
    "Expiration": "expiry-date=\"Fri, 06 Jan 2017 00:00:00 GMT\", rule-id=\"Rule for the Entire Bucket\""
}

To verify that the my-catalog.xml file was successfully uploaded, use the following command:

$ aws s3 ls s3://cdf-bucket/catalogs/ --region us-east-1

This command returns a list of all catalog files currently found in the bucket:

2015-12-07 15:02:17      10236 my-catalog.xml
2015-12-01 15:10:28        166 other-catalog.xml

Automating the Validation and Upload Process

You can usually write a script that will automate the validation and upload process. HAQM strongly recommends validating your catalog file before attempting to upload. If you create a script, please include a validation step as part of your upload process.

Additional Resources for AWS

These links provide further information on AWS and HAQM S3.

• HAQM Web Services: All AWS products
• HAQM Simple Storage Service (S3): Secure object storage
• HAQM Identity and Access Management (IAM): Define users and roles for your AWS account
• Start Developing with AWS: Documentation, SDKs, and sample apps for various languages and platforms
• AWS Command Line Interface (CLI): Simple access to AWS services
• Tools for HAQM Web Services: All SDKs and sample code
• AWS SDK For Java: Integrate AWS with your Java apps
• HAQM S3 Client: Java class for accessing HAQM S3
• Upload an Object Using the AWS SDK for Java: Sample code for uploading your data to HAQM S3

Catalog Uploading FAQ

Q: What's the maximum frequency at which we can publish a catalog file to our AWS S3 bucket?

A: HAQM does not have limits on publishing frequency; we adjust our ingestion pipeline as appropriate for the volume of uploaded files.

Q: After uploading my catalog file to my S3 bucket, how soon can I expect it to be ingested?

A: HAQM retrieves new catalog file uploads from partner S3 buckets every four hours. If your catalog file can be successfully ingested, it will be ingested at this time. If your catalog file fails ingestion, you'll need to fix your file, re-upload, and wait for the next four hour pickup window. If your catalog update must be faster than this, please discuss this with your HAQM representative. Note that this four-hour pickup window is subject to change.

Q: What is the delay between a catalog file upload and the content being available on the Fire TV device?

A: Typically, content will be available to customers within 2-4 hours after the catalog is ingested; however, due to possible caching and other conditions, content may take as long as 72 hours to become available for some apps.

Q: Do we need to upload a full catalog file every time we update? Can we upload a catalog file that only includes new and updated data?

A: You need to provide the full catalog file with each upload. HAQM uses the full file to calculate what needs to be deleted. Additionally, having the full file helps ensure that HAQM's catalog will not diverge from yours over time.

Q: What if we send several catalog updates within quick succession of each other? What happens if we produce catalog updates faster than HAQM can process them?

A: If you upload your catalog files faster than HAQM can process them, HAQM will just ignore the older copies and use the most recent file version. In other words, you cannot overload HAQM's system by uploading an abundance of catalogs to the S3 system.

Q: Do I need to rename my catalog file every time that I upload a new version of it to S3? How does HAQM determine which catalog file to use if my S3 bucket contains multiple files?

A: HAQM uses timestamps to determine the newest version of your catalog file and will always ingest and use the most recent version.

Troubleshooting

The following are common errors when uploading catalog files.

"AccessDenied" response when when calling the PutObject operation.

Check two things:

• Make sure that when you upload your catalog file via the aws s3api put-object command, you include the --acl bucket-owner-full-control parameter.
• Make sure you set up your IAM user S3 bucket permissions properly. Review the section Grant IAM Users Access to Your HAQM S3 Bucket (this section appears in Step D in Step 3: Set Up Your AWS Account for Catalog Ingestion).

"InvalidAccessKeyId" response when calling the PutObject operation.

The AWS Access Key Id is no longer be valid. Check your IAM user for new access keys. Review the section Grant IAM Users Access to Your HAQM S3 Bucket (this section appears in Step D in Step 3: Set Up Your AWS Account for Catalog Ingestion).

Next Steps

Now that you've uploaded your catalog file, go to the next step: Step 5: Verify Your Uploaded Catalog File.