Automating Oracle DMP files to OCI Buckets

Managing large volumes of Adhoc database export files across environments can quickly become a storage nightmare. As part of my continuous efforts to optimize Oracle DBA operations, I built a lightweight automation script to handle Oracle Data Pump dumps (.dmp, .DMP) and logs efficiently in Oracle Cloud Infrastructure (OCI). The Problem is DB dump folders such as /mnt/nfs mount often accumulate hundreds of .dmp and .log files. These take up terabytes of space, making it high costs on EBS file system. Goals is to automate to move Oracle expdp dmp files from ebs to OCI Buckets.

#!/bin/bash
# Author: Sandeep Reddy Narani
# Environment: Oracle Cloud Infrastructure (OCI)
# Description:
#   Uploads .dmp, .DMP, and .log files older than 15 days
#   from all subfolders under /mnt/dbadeeds (excluding DONT_DELETE)
#   to OCI Object Storage, deletes them locally upon successful upload,
#   and emails a summary report.

# ===== Configuration =====
BASE_DIR="/mnt/dbadeeds"
EXCLUDE_DIR="${BASE_DIR}/DONT_DELETE"

OCI_PATH="/mnt/dbadeeds/dba/oracle_oci/oci"
export OCI_CLI_CONFIG_FILE="/mnt/dbadeeds/dba/oracle_oci/config"
export PYTHONWARNINGS="ignore::DeprecationWarning"

BUCKET_NAME="OCIBUCKET-EXPDP-DMP"
PROFILE="DEFAULT"
LOG_DIR="/mnt/dbadeeds/dba"
LOG_FILE="${LOG_DIR}/oci_upload_$(date +%Y%m%d_%H%M).log"
EMAIL_TO="dbadeeds.com"
HOSTNAME=$(hostname)
DATE=$(date '+%Y-%m-%d %H:%M:%S')

# ===== Start =====
echo "========== OCI Upload Job Started: $DATE on $HOSTNAME ==========" | tee -a "$LOG_FILE"

# Check OCI binary
if [ ! -x "$OCI_PATH" ]; then
    echo "ERROR: OCI CLI not found or not executable at $OCI_PATH" | tee -a "$LOG_FILE"
    SUBJECT="[FAILED] OCI Upload - $HOSTNAME"
    echo "OCI CLI binary missing. Please verify OCI CLI installation." | mailx -s "$SUBJECT" "$EMAIL_TO" < "$LOG_FILE"
    exit 1
fi

# Get OCI namespace
NAMESPACE=$($OCI_PATH os ns get --profile "$PROFILE" --query 'data' --raw-output 2>>"$LOG_FILE")
if [ -z "$NAMESPACE" ]; then
    echo "ERROR: Failed to retrieve OCI namespace. Exiting." | tee -a "$LOG_FILE"
    SUBJECT="[FAILED] OCI Upload - $HOSTNAME"
    echo "Failed to retrieve OCI namespace." | mailx -s "$SUBJECT" "$EMAIL_TO" < "$LOG_FILE"
    exit 1
fi

SUCCESS_COUNT=0
FAIL_COUNT=0

# ===== Find Eligible Files (Exclude DONT_DELETE) =====
FILES=$(find "${BASE_DIR}" \
  -type f \( -iname "*.dmp" -o -iname "*.log" \) \
  -mtime +15 \
  -not -path "${EXCLUDE_DIR}/*")

if [ -z "$FILES" ]; then
    echo "No files found older than 15 days for upload (excluding DONT_DELETE)." | tee -a "$LOG_FILE"
else
    echo "Found files for upload (excluding DONT_DELETE):" | tee -a "$LOG_FILE"
    echo "$FILES" | tee -a "$LOG_FILE"
fi

# ===== Upload & Auto-Delete =====
echo "$FILES" | while read -r FILE; do
    [ -z "$FILE" ] && continue

    OBJECT_NAME=$(basename "$FILE")
    FOLDER_PATH=$(dirname "$FILE")
    RELATIVE_PATH=${FOLDER_PATH#/mnt/dbadeeds/}  # remove prefix
    OCI_OBJECT_PATH="${RELATIVE_PATH}/${OBJECT_NAME}"

    echo "Uploading $FILE to OCI path $OCI_OBJECT_PATH..." | tee -a "$LOG_FILE"

    if $OCI_PATH os object put \
        --namespace "$NAMESPACE" \
        --bucket-name "$BUCKET_NAME" \
        --file "$FILE" \
        --name "$OCI_OBJECT_PATH" \
        --profile "$PROFILE" >>"$LOG_FILE" 2>&1; then

        echo "✅ Upload successful: $OCI_OBJECT_PATH" | tee -a "$LOG_FILE"
        ((SUCCESS_COUNT++))

        if rm -f "$FILE"; then
            echo "🗑️ Deleted local file: $FILE" | tee -a "$LOG_FILE"
        else
            echo "⚠️ WARNING: Failed to delete local file: $FILE" | tee -a "$LOG_FILE"
        fi
    else
        echo "❌ Upload failed for: $FILE" | tee -a "$LOG_FILE"
        ((FAIL_COUNT++))
    fi
done

# ===== Log Rotation =====
find "$LOG_DIR" -name "oci_upload_*.log" -mtime +7 -exec gzip {} \; >/dev/null 2>&1

# ===== Email Summary =====
SUMMARY_FILE="${LOG_DIR}/oci_upload_summary_$(date +%Y%m%d_%H%M).txt"
{
  echo "OCI Upload Job Summary - $DATE"
  echo "--------------------------------------------"
  echo "Host           : $HOSTNAME"
  echo "Bucket Name    : $BUCKET_NAME"
  echo "Excluded Path  : $EXCLUDE_DIR"
  echo "Scanned Path   : $BASE_DIR"
  echo "Uploaded OK    : $SUCCESS_COUNT"
  echo "Failed Uploads : $FAIL_COUNT"
  echo
  echo "See detailed log: $LOG_FILE"
} > "$SUMMARY_FILE"

if [ $FAIL_COUNT -eq 0 ]; then
    SUBJECT="[SUCCESS] OCI Upload Completed - $HOSTNAME"
else
    SUBJECT="[PARTIAL/FAILED] OCI Upload Completed - $HOSTNAME"
fi

mailx -s "$SUBJECT" "$EMAIL_TO" < "$SUMMARY_FILE"

echo "Upload job completed. Summary emailed to $EMAIL_TO" | tee -a "$LOG_FILE"
echo "======================================================" | tee -a "$LOG_FILE"

The Impact

After implementing this automation, our OCI environments saw:

Over 80% cost reduction in local dump storage
Simplified cloud archival and compliance tracking
Zero manual intervention for daily or weekly cleanups

DBADEEDS