Unverified Commit c6ed850a authored by cmrd Senya's avatar cmrd Senya Committed by Benjamin Neff
Browse files

Memory usage optimization for archive export 

- Removed posts and non contacts from other's data
- Collections are exported in batches to lower memory footprint
- In base exporters create User object instead of keeping instance because it caches all associations

closes #7627
parent ea57fc5d

Changelog.md

+1 −0
Original line number Diff line number Diff line
@@ -12,6 +12,7 @@ 
* Use Bootstrap 3 progress-bar for polls [#7600](https://github.com/diaspora/diaspora/pull/7600)

* Enable frozen string literals [#7595](https://github.com/diaspora/diaspora/pull/7595)

* Remove `rails_admin_histories` table [#7597](https://github.com/diaspora/diaspora/pull/7597)

* Optimize memory usage on profile export [#7627](https://github.com/diaspora/diaspora/pull/7627)



## Bug fixes

* Fix displaying polls with long answers [#7579](https://github.com/diaspora/diaspora/pull/7579)

app/serializers/export/others_data_serializer.rb

+14 −15
Original line number Diff line number Diff line
@@ -4,33 +4,32 @@ module Export 
  class OthersDataSerializer < ActiveModel::Serializer

    # Relayables of other people in the archive: comments, likes, participations, poll participations where author is

    # the archive owner

    has_many :relayables, each_serializer: FederationEntitySerializer

    has_many :relayables, serializer: FlatMapArraySerializer, each_serializer: FederationEntitySerializer



    # Parent posts of user's own relayables. We have to save metadata to use

    # it in case when posts temporary unavailable on the target pod.

    has_many :posts, each_serializer: FederationEntitySerializer



    # Authors of posts where we participated and authors are not in contacts

    has_many :non_contact_authors, each_serializer: PersonMetadataSerializer

    def initialize(user_id)

      @user_id = user_id

      super(object)

    end



    private



    def object

      User.find(@user_id)

    end



    def relayables

      %i[comments likes poll_participations].map {|relayable|

        others_relayables.send(relayable)

      }.sum

        others_relayables.send(relayable).find_each(batch_size: 20)

      }

    end



    def others_relayables

      @others_relayables ||= Diaspora::Exporter::OthersRelayables.new(object.person_id)

    end



    def posts

      @posts ||= Diaspora::Exporter::PostsWithActivity.new(object).query

    end



    def non_contact_authors

      Diaspora::Exporter::NonContactAuthors.new(posts, object).query

    # Avoid calling pointless #embedded_in_root_associations method

    def serializable_data

      {}

    end

  end

end

app/serializers/export/own_post_serializer.rb

+1 −1
Original line number Diff line number Diff line
@@ -29,7 +29,7 @@ module Export 
    end



    def excluded_subscription_key

      entity.public ? :subscribed_users_ids : :subscribed_pods_uris

      object.public? ? :subscribed_users_ids : :subscribed_pods_uris

    end

  end

end

app/serializers/export/person_metadata_serializer.rb

deleted100644 → 0
+0 −19
Original line number Diff line number Diff line 
# frozen_string_literal: true



module Export

  class PersonMetadataSerializer < ActiveModel::Serializer

    attributes :guid,

               :account_id,

               :public_key



    private



    def account_id

      object.diaspora_handle

    end



    def public_key

      object.serialized_public_key

    end

  end

end

app/serializers/export/user_serializer.rb

+26 −2
Original line number Diff line number Diff line
@@ -18,12 +18,31 @@ module Export 
    has_many   :followed_tags

    has_many   :post_subscriptions



    has_many :relayables, each_serializer: Export::OwnRelayablesSerializer

    has_many :relayables, serializer: FlatMapArraySerializer, each_serializer: Export::OwnRelayablesSerializer



    def initialize(user_id, options={})

      @user_id = user_id

      super(object, options)

    end



    private



    def object

      User.find(@user_id)

    end



    def posts

      object.posts.find_each(batch_size: 20)

    end



    def contacts

      object.contacts.find_each(batch_size: 100)

    end



    def relayables

      [*comments, *likes, *poll_participations]

      [comments, likes, poll_participations].map {|relayable|

        relayable.find_each(batch_size: 20)

      }

    end



    %i[comments likes poll_participations].each {|collection|
@@ -47,5 +66,10 @@ module Export 
    def post_subscriptions

      Post.subscribed_by(object).pluck(:guid)

    end



    # Avoid calling pointless #embedded_in_root_associations method

    def serializable_data

      {}

    end

  end

end